[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:11 dist-test-slave-rpk1 krb5kdc[17199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:11 dist-test-slave-rpk1 krb5kdc[17199](info): set up 2 sockets
Sep 25 00:53:11 dist-test-slave-rpk1 krb5kdc[17199](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 00:53:12.161032 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:12.284648 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39895
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:12.396893 17212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:12.397205 17212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:12.397265 17212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:12.401450 17212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:12.401522 17212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:12.401549 17212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:12.401571 17212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:12.401597 17212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:12.407168 17212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:12.408327 17212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:12.409427 17212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:12.418243 17212 server_base.cc:1034] running on GCE node
W20260925 00:53:12.418418 17218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:12.418291 17217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:12.418262 17220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:12.419057 17212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:12.420158 17212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:12.421311 17212 hybrid_clock.cc:648] HybridClock initialized: now 1790297592421299 us; error 40 us; skew 500 ppm
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:12.423975 17212 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:12.424909 17212 webserver.cc:460] Webserver started at http://127.16.202.62:42065/ using document root <none> and password file <none>
I20260925 00:53:12.425439 17212 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:12.425505 17212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:12.425712 17212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:12.427734 17212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "4d44fedd36724ad78af68b70d293a966"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "438453b90112f6b862199598a7f4493b"
server_key_iv: "17132ac1818ff29a23821adaac76644f"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.428298 17212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "4d44fedd36724ad78af68b70d293a966"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "438453b90112f6b862199598a7f4493b"
server_key_iv: "17132ac1818ff29a23821adaac76644f"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.431936 17212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:53:12.434388 17227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.435606 17212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:12.435719 17212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "4d44fedd36724ad78af68b70d293a966"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "438453b90112f6b862199598a7f4493b"
server_key_iv: "17132ac1818ff29a23821adaac76644f"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.435818 17212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:12.463266 17212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:12.466920 17212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:12.467088 17212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:12.474037 17212 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39895
I20260925 00:53:12.474056 17279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39895 every 8 connection(s)
I20260925 00:53:12.475178 17212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:12.478266 17280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:12.482012 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17212
I20260925 00:53:12.482182 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:12.483860 17192 external_mini_cluster.cc:1428] Setting key 69ae79932b38dc924833bfb28dde6311
I20260925 00:53:12.484637 17280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966: Bootstrap starting.
I20260925 00:53:12.487656 17280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:12.488463 17280 log.cc:824] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:12.490361 17280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966: No bootstrap required, opened a new log
I20260925 00:53:12.494911 17280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d44fedd36724ad78af68b70d293a966" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39895 } }
I20260925 00:53:12.495160 17280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:12.495221 17280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d44fedd36724ad78af68b70d293a966, State: Initialized, Role: FOLLOWER
I20260925 00:53:12.495669 17280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d44fedd36724ad78af68b70d293a966" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39895 } }
I20260925 00:53:12.495841 17280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:12.495911 17280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:12.496002 17280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:12.497233 17280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d44fedd36724ad78af68b70d293a966" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39895 } }
I20260925 00:53:12.497612 17280 leader_election.cc:302] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d44fedd36724ad78af68b70d293a966; no voters:
I20260925 00:53:12.497943 17280 leader_election.cc:288] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:12.498065 17285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:12.498293 17285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [term 1 LEADER]: Becoming Leader. State: Replica: 4d44fedd36724ad78af68b70d293a966, State: Running, Role: LEADER
I20260925 00:53:12.498649 17285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d44fedd36724ad78af68b70d293a966" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39895 } }
I20260925 00:53:12.499311 17280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:12.499904 17286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d44fedd36724ad78af68b70d293a966. Latest consensus state: current_term: 1 leader_uuid: "4d44fedd36724ad78af68b70d293a966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d44fedd36724ad78af68b70d293a966" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39895 } } }
I20260925 00:53:12.500037 17286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:12.500667 17287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d44fedd36724ad78af68b70d293a966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d44fedd36724ad78af68b70d293a966" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39895 } } }
I20260925 00:53:12.500782 17287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966 [sys.catalog]: This master's current role is: LEADER
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
W20260925 00:53:12.504565 17300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:53:12.504643 17300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:53:12.504735 17291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:12.505628 17291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:12.510751 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:12.487500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.16.202.62:39895)
0925 00:53:12.488025 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:12.488043 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:12.489272 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 00:53:12.492631 (+ 3359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:12.492655 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:12.493105 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:12.493498 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:12.495738 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.497268 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.498998 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.499715 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.499889 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:12.504806 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:12.504813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:12.504831 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:12.504867 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:12.506631 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:12.507252 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:12.507257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:12.507262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:12.507412 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:12.507866 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:12.507870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:12.507871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:12.508459 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:12.508652 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:12.509438 (+ 786us) server_negotiation.cc:299] Negotiation successful
0925 00:53:12.509758 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":165,"threads_started":1}
I20260925 00:53:12.512429 17291 catalog_manager.cc:1347] Generated new cluster ID: eebfa04354e240a1ac0235fdf5d470ed
I20260925 00:53:12.512494 17291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:12.542109 17291 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:12.543246 17291 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:12.551071 17291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d44fedd36724ad78af68b70d293a966: Generated new TSK 0
I20260925 00:53:12.552063 17291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:12.591899 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39895
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:12.707435 17306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:12.707729 17306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:12.707777 17306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:12.711951 17306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:12.712042 17306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:12.712105 17306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:12.717751 17306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39895
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:12.718833 17306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:12.719949 17306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:12.729595 17311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:12.729648 17314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:12.729669 17306 server_base.cc:1034] running on GCE node
W20260925 00:53:12.729575 17312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:12.730185 17306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:12.730850 17306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:12.732018 17306 hybrid_clock.cc:648] HybridClock initialized: now 1790297592731993 us; error 36 us; skew 500 ppm
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:12.734782 17306 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:12.735761 17306 webserver.cc:460] Webserver started at http://127.16.202.1:45117/ using document root <none> and password file <none>
I20260925 00:53:12.736348 17306 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:12.736460 17306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:12.736734 17306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:12.738945 17306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "4100fdaaee534c82bc58a2bddfd6192e"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "4c97e148b41ec46f1f334ed8f210ae7d"
server_key_iv: "eb0392b7b28b1661de4f076e24266bf9"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.739513 17306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "4100fdaaee534c82bc58a2bddfd6192e"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "4c97e148b41ec46f1f334ed8f210ae7d"
server_key_iv: "eb0392b7b28b1661de4f076e24266bf9"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.743407 17306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 00:53:12.746016 17321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.747354 17306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:12.747519 17306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "4100fdaaee534c82bc58a2bddfd6192e"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "4c97e148b41ec46f1f334ed8f210ae7d"
server_key_iv: "eb0392b7b28b1661de4f076e24266bf9"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.747641 17306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:12.760941 17306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:12.763907 17306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:12.764097 17306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:12.764748 17306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:12.765626 17306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:12.765674 17306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.765727 17306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:12.765756 17306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.774710 17306 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:41965
I20260925 00:53:12.774766 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:41965 every 8 connection(s)
I20260925 00:53:12.775671 17306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:12.778486 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17306
I20260925 00:53:12.778609 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:12.778923 17192 external_mini_cluster.cc:1428] Setting key 66bdcb629e34ee45351964f2d83a8457
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:12.790747 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:12.777741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:58487 (local address 127.16.202.62:39895)
0925 00:53:12.777892 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:12.777896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:12.778972 (+ 1076us) server_negotiation.cc:407] Connection header received
0925 00:53:12.780473 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:12.780476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:12.780531 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:12.780603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:12.781887 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.782420 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.783832 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.784075 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.784167 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:12.787102 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:12.787105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:12.787107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:12.787135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:12.788504 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:12.789053 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:12.789056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:12.789058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:12.789139 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:12.789464 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:12.789468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:12.789470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:12.789722 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:12.789815 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:12.790348 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 00:53:12.790537 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:12.791577 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:12.778137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39895 (local address 127.16.202.1:58487)
0925 00:53:12.778804 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:12.778855 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:12.780201 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:12.780822 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:12.780831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:12.781214 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:12.781737 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:12.781753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.782545 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.782548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:12.783710 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:12.783722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.784247 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.784251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:12.784461 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:12.785105 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:12.785127 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:12.786907 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:12.788615 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:12.788622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:12.788641 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:12.788954 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:12.789223 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:12.789226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:12.789228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:12.789370 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:12.789823 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:12.789831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:12.790110 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 00:53:12.790279 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 00:53:12.790640 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":166,"threads_started":1}
I20260925 00:53:12.793123 17435 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39895
I20260925 00:53:12.793411 17435 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:12.794026 17435 heartbeater.cc:511] Master 127.16.202.62:39895 requested a full tablet report, sending...
I20260925 00:53:12.795755 17239 ts_manager.cc:194] Registered new tserver with Master: 4100fdaaee534c82bc58a2bddfd6192e (127.16.202.1:41965)
I20260925 00:53:12.797022 17239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:58487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:12.808247 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39895
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:12.921011 17440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:12.921298 17440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:12.921348 17440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:12.925477 17440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:12.925552 17440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:12.925640 17440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:12.931061 17440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39895
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:12.932231 17440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:12.933322 17440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:12.943214 17446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:12.943224 17448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:12.943224 17445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:12.943511 17440 server_base.cc:1034] running on GCE node
I20260925 00:53:12.943938 17440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:12.944595 17440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:12.945753 17440 hybrid_clock.cc:648] HybridClock initialized: now 1790297592945743 us; error 42 us; skew 500 ppm
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:12.948441 17440 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:12.949370 17440 webserver.cc:460] Webserver started at http://127.16.202.2:41465/ using document root <none> and password file <none>
I20260925 00:53:12.949929 17440 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:12.949995 17440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:12.950188 17440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:12.952363 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "545afeed0ce9473e8f40224aec38a86d"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "dcb017c15af943978723f4c200d0c0c0"
server_key_iv: "4aebc11ac371fa8889901cfbe398b678"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.952978 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "545afeed0ce9473e8f40224aec38a86d"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "dcb017c15af943978723f4c200d0c0c0"
server_key_iv: "4aebc11ac371fa8889901cfbe398b678"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.956801 17440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:12.959403 17455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.960733 17440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:12.960846 17440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "545afeed0ce9473e8f40224aec38a86d"
format_stamp: "Formatted at 2026-09-25 00:53:12 on dist-test-slave-rpk1"
server_key: "dcb017c15af943978723f4c200d0c0c0"
server_key_iv: "4aebc11ac371fa8889901cfbe398b678"
server_key_version: "encryptionkey@0"
I20260925 00:53:12.961002 17440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:12.974961 17440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:12.978263 17440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:12.978463 17440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:12.979111 17440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:12.979981 17440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:12.980034 17440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.980087 17440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:12.980115 17440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:12.989616 17440 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:39451
I20260925 00:53:12.989655 17568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:39451 every 8 connection(s)
I20260925 00:53:12.990586 17440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:12.994619 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17440
I20260925 00:53:12.994745 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:12.995011 17192 external_mini_cluster.cc:1428] Setting key f69a3deb70d369bdad09dee82afaeaea
Sep 25 00:53:12 dist-test-slave-rpk1 krb5kdc[17199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:13.004667 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:12.992392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:55713 (local address 127.16.202.62:39895)
0925 00:53:12.992544 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:12.992548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:12.993712 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 00:53:12.994597 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:12.994601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:12.994660 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:12.994759 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:12.996036 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.996536 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.997680 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.997899 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.997985 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:13.000841 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:13.000845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:13.000847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:13.000875 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:13.002444 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.003010 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.003013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.003014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.003092 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.003488 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.003491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.003492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.003729 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:13.003849 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:13.004366 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 00:53:13.004499 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:13.005590 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:12.992893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39895 (local address 127.16.202.2:55713)
0925 00:53:12.993559 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:12.993603 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:12.994385 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:12.994971 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:12.994979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:12.995395 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:12.995889 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:12.995902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.996653 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.996656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:12.997565 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:12.997573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:12.998038 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:12.998042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:12.998198 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:12.998798 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:12.998817 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:13.000636 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:13.002570 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:13.002577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:13.002596 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:13.002894 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:13.003174 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:13.003178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:13.003180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:13.003380 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:13.003842 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:13.003849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:13.004119 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 00:53:13.004297 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 00:53:13.004659 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:13.007445 17569 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39895
I20260925 00:53:13.007702 17569 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:13.008216 17569 heartbeater.cc:511] Master 127.16.202.62:39895 requested a full tablet report, sending...
I20260925 00:53:13.009361 17239 ts_manager.cc:194] Registered new tserver with Master: 545afeed0ce9473e8f40224aec38a86d (127.16.202.2:39451)
I20260925 00:53:13.010129 17239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:55713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:13.025715 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39895
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:13.142174 17574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:13.142459 17574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:13.142508 17574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:13.146687 17574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:13.146770 17574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:13.146845 17574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:13.152503 17574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39895
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:13.153628 17574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:13.154668 17574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:13.164631 17579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:13.164742 17580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:13.164816 17574 server_base.cc:1034] running on GCE node
W20260925 00:53:13.164752 17582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:13.165359 17574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:13.165977 17574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:13.167161 17574 hybrid_clock.cc:648] HybridClock initialized: now 1790297593167120 us; error 44 us; skew 500 ppm
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297593, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:13.170040 17574 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:13.170975 17574 webserver.cc:460] Webserver started at http://127.16.202.3:35931/ using document root <none> and password file <none>
I20260925 00:53:13.171535 17574 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:13.171602 17574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:13.171818 17574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:13.173965 17574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "c63b48a6227643e4af9b644651259949"
format_stamp: "Formatted at 2026-09-25 00:53:13 on dist-test-slave-rpk1"
server_key: "ff05bbed9bd88c2d2cd86763bc921f03"
server_key_iv: "59645cfdd10ce44ee5830a33d1699804"
server_key_version: "encryptionkey@0"
I20260925 00:53:13.174505 17574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "c63b48a6227643e4af9b644651259949"
format_stamp: "Formatted at 2026-09-25 00:53:13 on dist-test-slave-rpk1"
server_key: "ff05bbed9bd88c2d2cd86763bc921f03"
server_key_iv: "59645cfdd10ce44ee5830a33d1699804"
server_key_version: "encryptionkey@0"
I20260925 00:53:13.178262 17574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:13.180774 17589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:13.182058 17574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:13.182178 17574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "c63b48a6227643e4af9b644651259949"
format_stamp: "Formatted at 2026-09-25 00:53:13 on dist-test-slave-rpk1"
server_key: "ff05bbed9bd88c2d2cd86763bc921f03"
server_key_iv: "59645cfdd10ce44ee5830a33d1699804"
server_key_version: "encryptionkey@0"
I20260925 00:53:13.182286 17574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:13.198307 17574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:13.201673 17574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:13.201829 17574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:13.202418 17574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:13.203222 17574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:13.203267 17574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:13.203305 17574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:13.203325 17574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:13.212604 17574 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:34791
I20260925 00:53:13.212648 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:34791 every 8 connection(s)
I20260925 00:53:13.213557 17574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:13.222645 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17574
I20260925 00:53:13.222754 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297593, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:13.223009 17192 external_mini_cluster.cc:1428] Setting key d52f91c7b1f2a60706f24d4996b83529
I20260925 00:53:13.227317 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:13.215527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:53805 (local address 127.16.202.62:39895)
0925 00:53:13.215673 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:13.215677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:13.216742 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 00:53:13.217634 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:13.217636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:13.217683 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:13.217754 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:13.218941 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.219455 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.220701 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.220923 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.221000 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:13.223542 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:13.223545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:13.223547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:13.223574 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:13.225145 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.225693 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.225695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.225697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.225773 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.226122 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.226125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.226126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.226355 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:13.226453 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:13.226947 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 00:53:13.227048 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:53:13.227998 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:13.215930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39895 (local address 127.16.202.3:53805)
0925 00:53:13.216591 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:13.216634 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:13.217452 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:13.217913 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:13.217919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:13.218278 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:13.218806 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:13.218820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.219575 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.219579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:13.220581 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:13.220591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.221079 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.221082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:13.221246 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:13.221777 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:13.221793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:13.223337 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:13.225283 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:13.225290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:13.225309 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:13.225600 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:13.225852 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:13.225854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:13.225856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:13.226011 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:13.226450 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:13.226456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:13.226704 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 00:53:13.226868 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 00:53:13.227149 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":114,"threads_started":1}
I20260925 00:53:13.229053 17703 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39895
I20260925 00:53:13.229278 17703 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:13.229748 17703 heartbeater.cc:511] Master 127.16.202.62:39895 requested a full tablet report, sending...
I20260925 00:53:13.230805 17239 ts_manager.cc:194] Registered new tserver with Master: c63b48a6227643e4af9b644651259949 (127.16.202.3:34791)
I20260925 00:53:13.231383 17239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:53805
I20260925 00:53:13.237686 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:13.264508 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:13.254642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.16.202.1:41965)
0925 00:53:13.254968 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:13.254985 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:13.255012 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:53:13.255065 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:13.255069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:13.255309 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:13.255446 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:13.256148 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.257024 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.257880 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.258357 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.258457 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:13.260929 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:13.260940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:13.260955 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:13.260987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:13.262619 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.263074 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.263081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.263082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.263168 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.263431 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.263434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.263436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.263702 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:13.263928 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:13.264112 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 00:53:13.264313 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":115,"threads_started":1}
W20260925 00:53:13.265615 17349 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:13.291676 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:13.283481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.16.202.1:41965)
0925 00:53:13.283658 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:13.283662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:13.283733 (+ 71us) server_negotiation.cc:407] Connection header received
0925 00:53:13.283834 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:13.283837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:13.283881 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:13.283954 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:13.284801 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.285378 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.286315 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:13.286567 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:13.286653 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:13.288988 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:13.288991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:13.288993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:13.289022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:13.290228 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.290668 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.290671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.290673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.290756 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:13.290979 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:13.290981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:13.290982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:13.291138 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:13.291230 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:13.291412 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 00:53:13.291515 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 00:53:13.292761 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17306
I20260925 00:53:13.298151 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17440
I20260925 00:53:13.304787 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17574
I20260925 00:53:13.310084 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17212
2026-09-25T00:53:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2120 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17717](info): set up 2 sockets
Sep 25 00:53:13 dist-test-slave-rpk1 krb5kdc[17717](info): commencing operation
krb5kdc: starting...
W20260925 00:53:14.272351 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:14 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:14.396086 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38341
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:14.509423 17730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:14.509757 17730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:14.509927 17730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:14.514519 17730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:14.514626 17730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:14.514658 17730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:14.514680 17730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:14.514701 17730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:14.520533 17730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:38341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:14.521766 17730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:14.522833 17730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:14.531702 17738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:14.531692 17735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:14.531692 17736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:14.532207 17730 server_base.cc:1034] running on GCE node
I20260925 00:53:14.532862 17730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:14.534046 17730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:14.535246 17730 hybrid_clock.cc:648] HybridClock initialized: now 1790297594535228 us; error 29 us; skew 500 ppm
Sep 25 00:53:14 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297594, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:14.538340 17730 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:14.539364 17730 webserver.cc:460] Webserver started at http://127.16.202.62:34165/ using document root <none> and password file <none>
I20260925 00:53:14.539955 17730 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:14.540025 17730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:14.540313 17730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:14.542523 17730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "dd5e67f3d41f400e864098ceccd4f90b"
format_stamp: "Formatted at 2026-09-25 00:53:14 on dist-test-slave-rpk1"
server_key: "f2270da63a0f45a1b9ea61394bf5c57f"
server_key_iv: "bb463960cf2a105722dcfbb02773605c"
server_key_version: "encryptionkey@0"
I20260925 00:53:14.543118 17730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "dd5e67f3d41f400e864098ceccd4f90b"
format_stamp: "Formatted at 2026-09-25 00:53:14 on dist-test-slave-rpk1"
server_key: "f2270da63a0f45a1b9ea61394bf5c57f"
server_key_iv: "bb463960cf2a105722dcfbb02773605c"
server_key_version: "encryptionkey@0"
I20260925 00:53:14.547112 17730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:14.549666 17745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:14.550940 17730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:14.551059 17730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "dd5e67f3d41f400e864098ceccd4f90b"
format_stamp: "Formatted at 2026-09-25 00:53:14 on dist-test-slave-rpk1"
server_key: "f2270da63a0f45a1b9ea61394bf5c57f"
server_key_iv: "bb463960cf2a105722dcfbb02773605c"
server_key_version: "encryptionkey@0"
I20260925 00:53:14.551170 17730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:14.562845 17730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:14.566144 17730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:14.566334 17730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:14.573112 17730 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:38341
I20260925 00:53:14.573127 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:38341 every 8 connection(s)
I20260925 00:53:14.574092 17730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:14.577085 17798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:14.582599 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17730
I20260925 00:53:14.582731 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:14.582988 17192 external_mini_cluster.cc:1428] Setting key d80d278c10256f8b93c04b1361dfef55
I20260925 00:53:14.583477 17798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b: Bootstrap starting.
I20260925 00:53:14.586090 17798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:14.587000 17798 log.cc:824] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:14.589241 17798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b: No bootstrap required, opened a new log
Sep 25 00:53:14 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:14.593552 17798 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5e67f3d41f400e864098ceccd4f90b" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38341 } }
I20260925 00:53:14.593798 17798 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:14.593868 17798 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5e67f3d41f400e864098ceccd4f90b, State: Initialized, Role: FOLLOWER
I20260925 00:53:14.594419 17798 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5e67f3d41f400e864098ceccd4f90b" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38341 } }
I20260925 00:53:14.594579 17798 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:14.594645 17798 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:14.594745 17798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:14.595845 17798 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5e67f3d41f400e864098ceccd4f90b" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38341 } }
I20260925 00:53:14.596230 17798 leader_election.cc:302] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd5e67f3d41f400e864098ceccd4f90b; no voters:
I20260925 00:53:14.596591 17798 leader_election.cc:288] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:14.596746 17803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:14.596979 17803 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [term 1 LEADER]: Becoming Leader. State: Replica: dd5e67f3d41f400e864098ceccd4f90b, State: Running, Role: LEADER
I20260925 00:53:14.597405 17803 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5e67f3d41f400e864098ceccd4f90b" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38341 } }
I20260925 00:53:14.597864 17798 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:14.598743 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:14.584478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.16.202.62:38341)
0925 00:53:14.584970 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:14.584987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:14.585023 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:53:14.585734 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:14.585757 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:14.586114 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:14.586440 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:14.587328 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:14.588333 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:14.589467 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:14.589979 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:14.590116 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:14.592981 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:14.592987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:14.593003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:14.593032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:14.594977 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:14.595400 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:14.595406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:14.595412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:14.595565 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:14.595797 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:14.595800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:14.595802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:14.596341 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:14.596542 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:14.597342 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 00:53:14.597664 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":170,"threads_started":1}
I20260925 00:53:14.598760 17803 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5e67f3d41f400e864098ceccd4f90b. Latest consensus state: current_term: 1 leader_uuid: "dd5e67f3d41f400e864098ceccd4f90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5e67f3d41f400e864098ceccd4f90b" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38341 } } }
I20260925 00:53:14.598958 17803 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:14.599143 17804 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd5e67f3d41f400e864098ceccd4f90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5e67f3d41f400e864098ceccd4f90b" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38341 } } }
I20260925 00:53:14.599248 17804 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:14.599402 17812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:14.602665 17812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:14.610200 17812 catalog_manager.cc:1347] Generated new cluster ID: ff18951fd70c46949468f7025cbf8d62
I20260925 00:53:14.610278 17812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:14.622506 17812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:14.623525 17812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:14.633476 17812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd5e67f3d41f400e864098ceccd4f90b: Generated new TSK 0
I20260925 00:53:14.634380 17812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:14.671120 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38341
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:14.792740 17824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:14.792989 17824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:14.793031 17824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:14.796913 17824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:14.796970 17824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:14.797047 17824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:14.802397 17824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:14.803486 17824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:14.804504 17824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:14.814219 17830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:14.814224 17829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:14.814224 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:14.814427 17824 server_base.cc:1034] running on GCE node
I20260925 00:53:14.814960 17824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:14.815656 17824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:14.816818 17824 hybrid_clock.cc:648] HybridClock initialized: now 1790297594816801 us; error 30 us; skew 500 ppm
Sep 25 00:53:14 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297594, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:14.819985 17824 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:14.821022 17824 webserver.cc:460] Webserver started at http://127.16.202.1:39565/ using document root <none> and password file <none>
I20260925 00:53:14.821617 17824 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:14.821691 17824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:14.821914 17824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:14.824069 17824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "45d5367911014dc8ac20fd5308d60d56"
format_stamp: "Formatted at 2026-09-25 00:53:14 on dist-test-slave-rpk1"
server_key: "152b6265865fe61748f2b2b34cdc1d84"
server_key_iv: "fc6f301d4776418661d6049d898c5b2e"
server_key_version: "encryptionkey@0"
I20260925 00:53:14.824653 17824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "45d5367911014dc8ac20fd5308d60d56"
format_stamp: "Formatted at 2026-09-25 00:53:14 on dist-test-slave-rpk1"
server_key: "152b6265865fe61748f2b2b34cdc1d84"
server_key_iv: "fc6f301d4776418661d6049d898c5b2e"
server_key_version: "encryptionkey@0"
I20260925 00:53:14.828545 17824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:53:14.831175 17839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:14.832466 17824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:14.832597 17824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "45d5367911014dc8ac20fd5308d60d56"
format_stamp: "Formatted at 2026-09-25 00:53:14 on dist-test-slave-rpk1"
server_key: "152b6265865fe61748f2b2b34cdc1d84"
server_key_iv: "fc6f301d4776418661d6049d898c5b2e"
server_key_version: "encryptionkey@0"
I20260925 00:53:14.832746 17824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:14.852473 17824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:14.855304 17824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:14.855479 17824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:14.856042 17824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:14.856868 17824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:14.856916 17824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:14.856969 17824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:14.856993 17824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:14.865240 17824 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:44947
I20260925 00:53:14.865284 17952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:44947 every 8 connection(s)
I20260925 00:53:14.866110 17824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:14.868098 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17824
I20260925 00:53:14.868182 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:14.868412 17192 external_mini_cluster.cc:1428] Setting key 3f01484fac75cc3d62d8989966f637ae
Sep 25 00:53:14 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297594, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:14.879902 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:14.867803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:33507 (local address 127.16.202.62:38341)
0925 00:53:14.867962 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:14.867967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:14.868857 (+ 890us) server_negotiation.cc:407] Connection header received
0925 00:53:14.870253 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:14.870256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:14.870307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:14.870410 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:14.871603 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:14.872128 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:14.873315 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:14.873544 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:14.873633 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:14.876366 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:14.876368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:14.876371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:14.876396 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:14.877790 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:14.878328 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:14.878332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:14.878334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:14.878412 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:14.878725 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:14.878729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:14.878731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:14.878954 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:14.879043 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:14.879596 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 00:53:14.879733 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:53:14.880761 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:14.868237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38341 (local address 127.16.202.1:33507)
0925 00:53:14.868699 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:14.868751 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:14.870055 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:14.870542 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:14.870548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:14.870897 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:14.871456 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:14.871470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:14.872256 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:14.872259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:14.873192 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:14.873210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:14.873698 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:14.873703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:14.873860 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:14.874457 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:14.874474 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:14.876183 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:14.877900 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:14.877907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:14.877923 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:14.878230 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:14.878490 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:14.878494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:14.878496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:14.878631 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:14.879051 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:14.879058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:14.879363 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 00:53:14.879536 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 00:53:14.879867 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260925 00:53:14.882130 17953 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38341
I20260925 00:53:14.882368 17953 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:14.882856 17953 heartbeater.cc:511] Master 127.16.202.62:38341 requested a full tablet report, sending...
I20260925 00:53:14.884470 17762 ts_manager.cc:194] Registered new tserver with Master: 45d5367911014dc8ac20fd5308d60d56 (127.16.202.1:44947)
I20260925 00:53:14.885672 17762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:33507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:14.895687 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38341
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:15.013270 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:15.013571 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:15.013617 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:15.017753 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:15.017830 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:15.017899 17958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:15.023380 17958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:15.024442 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:15.025748 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:15.036917 17958 server_base.cc:1034] running on GCE node
W20260925 00:53:15.036861 17963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:15.036861 17964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:15.036862 17966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:15.037650 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:15.038316 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:15.039506 17958 hybrid_clock.cc:648] HybridClock initialized: now 1790297595039490 us; error 25 us; skew 500 ppm
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:15.042430 17958 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:15.043329 17958 webserver.cc:460] Webserver started at http://127.16.202.2:38333/ using document root <none> and password file <none>
I20260925 00:53:15.043893 17958 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:15.043956 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:15.044147 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:15.046356 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "adb37126a086412889800ddce616011a"
format_stamp: "Formatted at 2026-09-25 00:53:15 on dist-test-slave-rpk1"
server_key: "566fafba613d472bcb72aa549fa19e96"
server_key_iv: "772e35dd86747991f14d2ffbee5dbf45"
server_key_version: "encryptionkey@0"
I20260925 00:53:15.046921 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "adb37126a086412889800ddce616011a"
format_stamp: "Formatted at 2026-09-25 00:53:15 on dist-test-slave-rpk1"
server_key: "566fafba613d472bcb72aa549fa19e96"
server_key_iv: "772e35dd86747991f14d2ffbee5dbf45"
server_key_version: "encryptionkey@0"
I20260925 00:53:15.050788 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 00:53:15.053323 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:15.054867 17958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:15.054998 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "adb37126a086412889800ddce616011a"
format_stamp: "Formatted at 2026-09-25 00:53:15 on dist-test-slave-rpk1"
server_key: "566fafba613d472bcb72aa549fa19e96"
server_key_iv: "772e35dd86747991f14d2ffbee5dbf45"
server_key_version: "encryptionkey@0"
I20260925 00:53:15.055109 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:15.084841 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:15.087698 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:15.087874 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:15.088454 17958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:15.089318 17958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:15.089371 17958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:15.089424 17958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:15.089450 17958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:15.097986 17958 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:45545
I20260925 00:53:15.098006 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:45545 every 8 connection(s)
I20260925 00:53:15.099066 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:15.102452 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 17958
I20260925 00:53:15.102551 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:15.102780 17192 external_mini_cluster.cc:1428] Setting key 7c4585904b176d01e158807eb58bb4bc
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:15.114251 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.100978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:55305 (local address 127.16.202.62:38341)
0925 00:53:15.101120 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.101125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.102083 (+ 958us) server_negotiation.cc:407] Connection header received
0925 00:53:15.103444 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.103447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.103490 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.103572 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:15.104792 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.105521 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.107133 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.107355 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.107437 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.110385 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:15.110389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:15.110391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:15.110419 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:15.111820 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.112397 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.112401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.112402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.112504 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.113021 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.113024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.113025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.113247 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:15.113345 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.113911 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.114073 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:15.115223 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.101424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38341 (local address 127.16.202.2:55305)
0925 00:53:15.101928 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.101975 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.103267 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.103727 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.103734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.104107 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:15.104614 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.104628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.105657 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.105661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.107004 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.107014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.107502 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.107506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.107690 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.108379 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:15.108399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:15.110183 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:15.111961 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:15.111968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:15.111988 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:15.112256 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:15.112621 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:15.112624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:15.112625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:15.112863 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:15.113356 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:15.113363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:15.113645 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.113823 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.114203 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260925 00:53:15.116850 18087 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38341
I20260925 00:53:15.117120 18087 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:15.117707 18087 heartbeater.cc:511] Master 127.16.202.62:38341 requested a full tablet report, sending...
I20260925 00:53:15.118854 17762 ts_manager.cc:194] Registered new tserver with Master: adb37126a086412889800ddce616011a (127.16.202.2:45545)
I20260925 00:53:15.119453 17762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:55305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:15.131867 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38341
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:15.245337 18092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:15.245631 18092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:15.245676 18092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:15.249830 18092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:15.249897 18092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:15.249959 18092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:15.255438 18092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:15.256501 18092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:15.257609 18092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:15.268209 18098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:15.268244 18100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:15.268226 18097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:15.268714 18092 server_base.cc:1034] running on GCE node
I20260925 00:53:15.269277 18092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:15.269994 18092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:15.271166 18092 hybrid_clock.cc:648] HybridClock initialized: now 1790297595271137 us; error 47 us; skew 500 ppm
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:15.274055 18092 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:15.275027 18092 webserver.cc:460] Webserver started at http://127.16.202.3:43407/ using document root <none> and password file <none>
I20260925 00:53:15.275633 18092 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:15.275705 18092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:15.275905 18092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:15.278163 18092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "5c411f45cabc4d0f95edafc243b379f1"
format_stamp: "Formatted at 2026-09-25 00:53:15 on dist-test-slave-rpk1"
server_key: "0945c0f74a6a5f429284011af63f8b24"
server_key_iv: "ab827cbb2006bb868185b35af26b18ec"
server_key_version: "encryptionkey@0"
I20260925 00:53:15.278796 18092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "5c411f45cabc4d0f95edafc243b379f1"
format_stamp: "Formatted at 2026-09-25 00:53:15 on dist-test-slave-rpk1"
server_key: "0945c0f74a6a5f429284011af63f8b24"
server_key_iv: "ab827cbb2006bb868185b35af26b18ec"
server_key_version: "encryptionkey@0"
I20260925 00:53:15.282894 18092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:15.285704 18107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:15.287191 18092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:15.287341 18092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "5c411f45cabc4d0f95edafc243b379f1"
format_stamp: "Formatted at 2026-09-25 00:53:15 on dist-test-slave-rpk1"
server_key: "0945c0f74a6a5f429284011af63f8b24"
server_key_iv: "ab827cbb2006bb868185b35af26b18ec"
server_key_version: "encryptionkey@0"
I20260925 00:53:15.287463 18092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:15.320253 18092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:15.323704 18092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:15.323906 18092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:15.324522 18092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:15.325428 18092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:15.325474 18092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:15.325526 18092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:15.325553 18092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:15.334582 18092 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:41107
I20260925 00:53:15.334606 18220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:41107 every 8 connection(s)
I20260925 00:53:15.335559 18092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:15.339008 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18092
I20260925 00:53:15.339146 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:15.339516 17192 external_mini_cluster.cc:1428] Setting key 236feadd60407568b8ae2b30dc15a10e
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:15.350746 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.337687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:49309 (local address 127.16.202.62:38341)
0925 00:53:15.337842 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.337845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.338799 (+ 954us) server_negotiation.cc:407] Connection header received
0925 00:53:15.340578 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.340581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.340645 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.340768 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:15.342274 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.342880 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.344091 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.344322 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.344418 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.347167 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:15.347170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:15.347173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:15.347205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:15.348556 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.349154 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.349157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.349159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.349248 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.349599 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.349602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.349604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.349807 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:15.349894 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.350435 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.350564 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:53:15.351604 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.338094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38341 (local address 127.16.202.3:49309)
0925 00:53:15.338587 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.338641 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.340316 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.340957 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.340964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.341498 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:15.342119 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.342134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.343021 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.343024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.343957 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.343969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.344485 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.344490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.344702 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.345356 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:15.345374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:15.346960 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:15.348729 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:15.348736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:15.348754 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:15.349047 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:15.349343 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:15.349346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:15.349348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:15.349499 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:15.349908 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:15.349916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:15.350185 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.350378 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.350726 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":116,"threads_started":1}
I20260925 00:53:15.352991 18221 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38341
I20260925 00:53:15.353331 18221 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:15.353885 18221 heartbeater.cc:511] Master 127.16.202.62:38341 requested a full tablet report, sending...
I20260925 00:53:15.354912 17762 ts_manager.cc:194] Registered new tserver with Master: 5c411f45cabc4d0f95edafc243b379f1 (127.16.202.3:41107)
I20260925 00:53:15.355535 17762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:49309
I20260925 00:53:15.365871 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:53:15.377220 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.369578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.16.202.62:38341)
0925 00:53:15.369696 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.369699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.369905 (+ 206us) server_negotiation.cc:407] Connection header received
0925 00:53:15.370015 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.370018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.370064 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.370125 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:15.370825 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.371349 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.372262 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.372479 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.372554 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.374075 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:15.374078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:15.374080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:15.374110 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:15.375470 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.375996 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.375999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.376001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.376088 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.376378 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.376380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.376381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.376604 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:15.376735 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.376973 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.377077 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:53:15.382616 17762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:15.385735 17762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:15.401065 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.394586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:45545 (local address 127.0.0.1:32896)
0925 00:53:15.394982 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.395041 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.395169 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.396525 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.396531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.396558 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:15.396749 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.396758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.398191 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.398195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.399447 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.399457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.400417 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.400421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.400533 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.400574 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.400738 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.400883 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":93,"thread_start_us":73,"threads_started":1}
I20260925 00:53:15.401069 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.395516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.16.202.2:45545)
0925 00:53:15.395957 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.395967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.395989 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:53:15.396064 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.396069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.396313 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.396502 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:15.396993 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.398000 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.399598 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.400284 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.400401 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.400572 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.400777 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.400883 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":125,"threads_started":1}
I20260925 00:53:15.401261 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.394586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41107 (local address 127.0.0.1:36110)
0925 00:53:15.395573 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.395588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.395684 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.396732 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.396735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.396754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:15.396856 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.396863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.398146 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.398149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.399538 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.399547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.400849 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.400853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.400964 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.400986 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.401053 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.401121 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":75,"thread_start_us":105,"threads_started":1}
I20260925 00:53:15.401712 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.395892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.16.202.3:41107)
0925 00:53:15.396230 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.396243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.396265 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:53:15.396336 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.396340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.396536 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.396704 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:15.396964 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.397995 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.399665 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.400490 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.400590 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.400788 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.401450 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.401558 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":143,"threads_started":1}
I20260925 00:53:15.402136 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.394838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44947 (local address 127.0.0.1:50446)
0925 00:53:15.397069 (+ 2231us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.397083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.397192 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.397713 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.397716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.397737 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:15.397844 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.397851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.399364 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.399367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.400476 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.400488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.401754 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.401757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.401846 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.401866 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.401940 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.402013 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2164,"thread_start_us":113,"threads_started":1}
I20260925 00:53:15.402254 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.395566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.16.202.1:44947)
0925 00:53:15.395956 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.395964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.397200 (+ 1236us) server_negotiation.cc:407] Connection header received
0925 00:53:15.397296 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.397301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.397573 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.397720 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:15.397957 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.399242 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.400620 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.401450 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.401526 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.401657 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.401993 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.402099 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":80,"threads_started":1}
I20260925 00:53:15.404551 18155 tablet_service.cc:1467] Processing CreateTablet for tablet f9b0f00b3fde41bcb7d9601609ecd64c (DEFAULT_TABLE table=test-table [id=c880f4dc1caf4fa5be06aff47b40393f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:15.404564 18021 tablet_service.cc:1467] Processing CreateTablet for tablet f9b0f00b3fde41bcb7d9601609ecd64c (DEFAULT_TABLE table=test-table [id=c880f4dc1caf4fa5be06aff47b40393f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:15.405639 18155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b0f00b3fde41bcb7d9601609ecd64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:15.405686 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b0f00b3fde41bcb7d9601609ecd64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:15.405464 17887 tablet_service.cc:1467] Processing CreateTablet for tablet f9b0f00b3fde41bcb7d9601609ecd64c (DEFAULT_TABLE table=test-table [id=c880f4dc1caf4fa5be06aff47b40393f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:15.406498 17887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b0f00b3fde41bcb7d9601609ecd64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:15.411960 18238 tablet_bootstrap.cc:492] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: Bootstrap starting.
I20260925 00:53:15.414140 18239 tablet_bootstrap.cc:492] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: Bootstrap starting.
I20260925 00:53:15.414621 18240 tablet_bootstrap.cc:492] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: Bootstrap starting.
I20260925 00:53:15.414800 18238 tablet_bootstrap.cc:654] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:15.416203 18238 log.cc:824] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:15.416492 18239 tablet_bootstrap.cc:654] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:15.416836 18240 tablet_bootstrap.cc:654] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:15.417402 18239 log.cc:824] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:15.418166 18240 log.cc:824] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:15.418833 18238 tablet_bootstrap.cc:492] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: No bootstrap required, opened a new log
I20260925 00:53:15.419059 18238 ts_tablet_manager.cc:1397] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:53:15.419463 18239 tablet_bootstrap.cc:492] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: No bootstrap required, opened a new log
I20260925 00:53:15.419682 18239 ts_tablet_manager.cc:1397] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:53:15.420495 18240 tablet_bootstrap.cc:492] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: No bootstrap required, opened a new log
I20260925 00:53:15.420697 18240 ts_tablet_manager.cc:1397] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 00:53:15.422719 18239 raft_consensus.cc:348] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.422977 18239 raft_consensus.cc:374] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:15.423055 18239 raft_consensus.cc:729] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb37126a086412889800ddce616011a, State: Initialized, Role: FOLLOWER
I20260925 00:53:15.423112 18238 raft_consensus.cc:348] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.423390 18238 raft_consensus.cc:374] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:15.423486 18238 raft_consensus.cc:729] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c411f45cabc4d0f95edafc243b379f1, State: Initialized, Role: FOLLOWER
I20260925 00:53:15.423552 18239 consensus_queue.cc:260] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.423960 18238 consensus_queue.cc:260] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.424762 18239 ts_tablet_manager.cc:1428] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:53:15.425124 18087 heartbeater.cc:503] Master 127.16.202.62:38341 was elected leader, sending a full tablet report...
I20260925 00:53:15.425042 18240 raft_consensus.cc:348] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.425283 18240 raft_consensus.cc:374] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:15.425326 18240 raft_consensus.cc:729] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d5367911014dc8ac20fd5308d60d56, State: Initialized, Role: FOLLOWER
I20260925 00:53:15.425750 18240 consensus_queue.cc:260] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.426332 18238 ts_tablet_manager.cc:1428] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:53:15.426429 17953 heartbeater.cc:503] Master 127.16.202.62:38341 was elected leader, sending a full tablet report...
I20260925 00:53:15.426705 18240 ts_tablet_manager.cc:1428] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 00:53:15.426738 18221 heartbeater.cc:503] Master 127.16.202.62:38341 was elected leader, sending a full tablet report...
W20260925 00:53:15.586613 18222 tablet.cc:2369] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:15.600427 18088 tablet.cc:2369] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:15.617453 17954 tablet.cc:2369] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:15.624563 18246 raft_consensus.cc:482] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:15.624837 18246 raft_consensus.cc:504] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.626027 18246 leader_election.cc:288] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adb37126a086412889800ddce616011a (127.16.202.2:45545), 5c411f45cabc4d0f95edafc243b379f1 (127.16.202.3:41107)
I20260925 00:53:15.629911 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.626416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34575 (local address 127.16.202.3:41107)
0925 00:53:15.626562 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.626567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.626889 (+ 322us) server_negotiation.cc:407] Connection header received
0925 00:53:15.626990 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.626993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.627039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.627102 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:15.627352 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.627828 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.628954 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.629349 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.629453 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.629505 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.629723 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.629790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:53:15.629922 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.626439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:37883 (local address 127.16.202.2:45545)
0925 00:53:15.626561 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.626566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.626796 (+ 230us) server_negotiation.cc:407] Connection header received
0925 00:53:15.626923 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.626925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.626967 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.627028 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:15.627283 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.627777 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.628962 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.629347 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.629452 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.629505 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.629727 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.629799 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:15.629948 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.626335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:45545 (local address 127.16.202.1:37883)
0925 00:53:15.626709 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.626720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.626827 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.627043 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.627045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.627059 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:15.627189 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.627194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.627876 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.627879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.628848 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.628857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.629452 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.629456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.629625 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.629643 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.629711 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.629786 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":89,"thread_start_us":114,"threads_started":1}
I20260925 00:53:15.630201 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.626335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41107 (local address 127.16.202.1:34575)
0925 00:53:15.626789 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:15.626800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:15.626908 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:15.627119 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:15.627122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:15.627136 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:15.627247 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.627253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.627931 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.627934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.628844 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:15.628854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.629454 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.629457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:15.629613 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.629636 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:15.629706 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:53:15.629782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":75,"thread_start_us":113,"threads_started":1}
I20260925 00:53:15.630592 18041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b0f00b3fde41bcb7d9601609ecd64c" candidate_uuid: "45d5367911014dc8ac20fd5308d60d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb37126a086412889800ddce616011a" is_pre_election: true
I20260925 00:53:15.630805 18175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b0f00b3fde41bcb7d9601609ecd64c" candidate_uuid: "45d5367911014dc8ac20fd5308d60d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c411f45cabc4d0f95edafc243b379f1" is_pre_election: true
I20260925 00:53:15.630887 18041 raft_consensus.cc:2454] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d5367911014dc8ac20fd5308d60d56 in term 0.
I20260925 00:53:15.631062 18175 raft_consensus.cc:2454] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d5367911014dc8ac20fd5308d60d56 in term 0.
I20260925 00:53:15.631340 17842 leader_election.cc:302] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d5367911014dc8ac20fd5308d60d56, adb37126a086412889800ddce616011a; no voters:
I20260925 00:53:15.631608 18246 raft_consensus.cc:2781] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:15.631683 18246 raft_consensus.cc:482] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:15.631733 18246 raft_consensus.cc:3037] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:15.633198 18246 raft_consensus.cc:504] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.633538 18246 leader_election.cc:288] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [CANDIDATE]: Term 1 election: Requested vote from peers adb37126a086412889800ddce616011a (127.16.202.2:45545), 5c411f45cabc4d0f95edafc243b379f1 (127.16.202.3:41107)
I20260925 00:53:15.633926 18175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b0f00b3fde41bcb7d9601609ecd64c" candidate_uuid: "45d5367911014dc8ac20fd5308d60d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c411f45cabc4d0f95edafc243b379f1"
I20260925 00:53:15.633922 18041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b0f00b3fde41bcb7d9601609ecd64c" candidate_uuid: "45d5367911014dc8ac20fd5308d60d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb37126a086412889800ddce616011a"
I20260925 00:53:15.634030 18175 raft_consensus.cc:3037] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:15.634042 18041 raft_consensus.cc:3037] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:15.634975 18175 raft_consensus.cc:2454] T f9b0f00b3fde41bcb7d9601609ecd64c P 5c411f45cabc4d0f95edafc243b379f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d5367911014dc8ac20fd5308d60d56 in term 1.
I20260925 00:53:15.635039 18041 raft_consensus.cc:2454] T f9b0f00b3fde41bcb7d9601609ecd64c P adb37126a086412889800ddce616011a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d5367911014dc8ac20fd5308d60d56 in term 1.
I20260925 00:53:15.635239 17841 leader_election.cc:302] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d5367911014dc8ac20fd5308d60d56, 5c411f45cabc4d0f95edafc243b379f1; no voters:
I20260925 00:53:15.635401 18246 raft_consensus.cc:2781] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:15.635632 18246 raft_consensus.cc:686] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [term 1 LEADER]: Becoming Leader. State: Replica: 45d5367911014dc8ac20fd5308d60d56, State: Running, Role: LEADER
I20260925 00:53:15.635962 18246 consensus_queue.cc:237] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } }
I20260925 00:53:15.639497 17761 catalog_manager.cc:5495] T f9b0f00b3fde41bcb7d9601609ecd64c P 45d5367911014dc8ac20fd5308d60d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d5367911014dc8ac20fd5308d60d56 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "45d5367911014dc8ac20fd5308d60d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb37126a086412889800ddce616011a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 45545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c411f45cabc4d0f95edafc243b379f1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d5367911014dc8ac20fd5308d60d56" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44947 } health_report { overall_health: HEALTHY } } }
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:15.704376 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.695453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.16.202.1:44947)
0925 00:53:15.695633 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.695637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.695747 (+ 110us) server_negotiation.cc:407] Connection header received
0925 00:53:15.695909 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.695913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.695964 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.696040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:15.696816 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.697363 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.698263 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.698492 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.698583 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.701148 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:15.701164 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:15.701180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:15.701209 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:15.702798 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.703227 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.703233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.703234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.703325 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.703555 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.703557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.703558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.703800 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:15.703967 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.704117 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.704219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:15.730093 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.721951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.16.202.1:44947)
0925 00:53:15.722102 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.722106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.722266 (+ 160us) server_negotiation.cc:407] Connection header received
0925 00:53:15.722399 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.722402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.722447 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.722513 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:15.723213 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.723745 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.724555 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.724829 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.724934 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.727256 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:15.727259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:15.727261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:15.727286 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:15.728392 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.728857 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.728861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.728863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.728960 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.729272 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.729275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.729277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.729491 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:15.729608 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.729870 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.729971 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:15.755267 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:15.746777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.16.202.1:44947)
0925 00:53:15.746948 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:15.746951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:15.747022 (+ 71us) server_negotiation.cc:407] Connection header received
0925 00:53:15.747171 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:15.747173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:15.747219 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:15.747286 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:15.748014 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.748545 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.749429 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:15.749640 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:15.749717 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:15.752260 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:15.752263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:15.752265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:15.752291 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:15.753579 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.754079 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.754084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.754085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.754172 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:15.754439 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:15.754442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:15.754444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:15.754637 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:15.754778 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:15.754996 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 00:53:15.755111 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 00:53:15.756649 17867 tablet_service.cc:2908] Rejecting scan request for tablet f9b0f00b3fde41bcb7d9601609ecd64c: Uninitialized: clean time has not yet been initialized
I20260925 00:53:15.759034 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17824
I20260925 00:53:15.764640 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17958
I20260925 00:53:15.769542 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18092
I20260925 00:53:15.774399 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 17730
2026-09-25T00:53:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2464 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[18263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[18263](info): set up 2 sockets
Sep 25 00:53:15 dist-test-slave-rpk1 krb5kdc[18263](info): commencing operation
krb5kdc: starting...
W20260925 00:53:16.736485 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:16 dist-test-slave-rpk1 krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:16.859200 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:46271
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:46271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:16.971318 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:16.971621 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:16.971663 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:16.975841 18276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:16.975919 18276 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 00:53:16.975937 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:16.975955 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:16.975970 18276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:16.975986 18276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:16.981834 18276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:46271
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:16.983062 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:16.984182 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:16.993192 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:16.993173 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:16.993306 18276 server_base.cc:1034] running on GCE node
W20260925 00:53:16.993145 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:16.994103 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:16.995302 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:16.996490 18276 hybrid_clock.cc:648] HybridClock initialized: now 1790297596996474 us; error 32 us; skew 500 ppm
Sep 25 00:53:16 dist-test-slave-rpk1 krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297596, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:17.003548 18276 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:17.004642 18276 webserver.cc:460] Webserver started at http://127.16.202.62:36963/ using document root <none> and password file <none>
I20260925 00:53:17.005326 18276 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:17.005401 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:17.005625 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:17.007885 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "0d3f74b54b2146529807076317568c27"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "0c205d96d88a895d6da84289db6e82bb"
server_key_iv: "4a282b4b6b1ac57bd3ba4cf3bf5ff1a6"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.008504 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "0d3f74b54b2146529807076317568c27"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "0c205d96d88a895d6da84289db6e82bb"
server_key_iv: "4a282b4b6b1ac57bd3ba4cf3bf5ff1a6"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.012539 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:53:17.015503 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.017022 18276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:17.017158 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "0d3f74b54b2146529807076317568c27"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "0c205d96d88a895d6da84289db6e82bb"
server_key_iv: "4a282b4b6b1ac57bd3ba4cf3bf5ff1a6"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.017279 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:17.039223 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:17.042508 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:17.042716 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:17.050987 18276 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:46271
I20260925 00:53:17.050998 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:46271 every 8 connection(s)
I20260925 00:53:17.052032 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:17.054962 18354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:17.056160 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18276
I20260925 00:53:17.056275 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:17.056528 17192 external_mini_cluster.cc:1428] Setting key 260a77bcf2a0a377478268a3f144a891
I20260925 00:53:17.061578 18354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: Bootstrap starting.
I20260925 00:53:17.064411 18354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:17.065492 18354 log.cc:824] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:17.067973 18354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: No bootstrap required, opened a new log
I20260925 00:53:17.071597 18354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3f74b54b2146529807076317568c27" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46271 } }
I20260925 00:53:17.071859 18354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:17.071926 18354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3f74b54b2146529807076317568c27, State: Initialized, Role: FOLLOWER
I20260925 00:53:17.072194 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.057982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.16.202.62:46271)
0925 00:53:17.058406 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.058422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.058459 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:53:17.059099 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.059118 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.059476 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.059768 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.060702 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.061741 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.062885 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.063410 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.063531 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.066655 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.066661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.066677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.066707 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.069101 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.069658 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.069664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.069669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.069791 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.070086 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.070088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.070089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.070541 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.070697 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.070981 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.071275 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260925 00:53:17.072427 18354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3f74b54b2146529807076317568c27" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46271 } }
I20260925 00:53:17.072566 18354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:17.072633 18354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:17.072752 18354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:17.073683 18354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3f74b54b2146529807076317568c27" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46271 } }
I20260925 00:53:17.074021 18354 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3f74b54b2146529807076317568c27; no voters:
I20260925 00:53:17.074326 18354 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:17.074420 18359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:17.074604 18359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3f74b54b2146529807076317568c27, State: Running, Role: LEADER
I20260925 00:53:17.074944 18359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3f74b54b2146529807076317568c27" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46271 } }
I20260925 00:53:17.075687 18354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:17.076723 18361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3f74b54b2146529807076317568c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3f74b54b2146529807076317568c27" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46271 } } }
I20260925 00:53:17.076789 18360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3f74b54b2146529807076317568c27. Latest consensus state: current_term: 1 leader_uuid: "0d3f74b54b2146529807076317568c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3f74b54b2146529807076317568c27" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46271 } } }
I20260925 00:53:17.076846 18361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:17.076887 18360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:17.077353 18368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:17.080649 18368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:17.085825 18368 catalog_manager.cc:1347] Generated new cluster ID: f6514d4efbf1415186caa2932b727ba5
I20260925 00:53:17.085911 18368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:17.091259 18368 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:17.092299 18368 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:17.099332 18368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: Generated new TSK 0
I20260925 00:53:17.100052 18368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:17.131183 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:46271
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:17.242141 18380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:17.242391 18380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:17.242435 18380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:17.246567 18380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:17.246624 18380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:17.246667 18380 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:17.246719 18380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:17.252004 18380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:17.253093 18380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:17.254122 18380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:17.264065 18380 server_base.cc:1034] running on GCE node
W20260925 00:53:17.264010 18385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:17.264019 18388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:17.264007 18386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:17.264752 18380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:17.265460 18380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:17.266604 18380 hybrid_clock.cc:648] HybridClock initialized: now 1790297597266582 us; error 35 us; skew 500 ppm
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:17.269629 18380 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:17.270630 18380 webserver.cc:460] Webserver started at http://127.16.202.1:33337/ using document root <none> and password file <none>
I20260925 00:53:17.271196 18380 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:17.271265 18380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:17.271487 18380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:17.273696 18380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "bf212653acac4518b7df4e302a4c8f7f"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "e6230dabfaae14fe09012d5b055c0680"
server_key_iv: "e1eaf85b88fee16a975d7ccd5aaf0bfb"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.274266 18380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "bf212653acac4518b7df4e302a4c8f7f"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "e6230dabfaae14fe09012d5b055c0680"
server_key_iv: "e1eaf85b88fee16a975d7ccd5aaf0bfb"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.278221 18380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:17.280802 18395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.282027 18380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:17.282138 18380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "bf212653acac4518b7df4e302a4c8f7f"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "e6230dabfaae14fe09012d5b055c0680"
server_key_iv: "e1eaf85b88fee16a975d7ccd5aaf0bfb"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.282239 18380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:17.307888 18380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:17.310781 18380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:17.310962 18380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:17.312449 18380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:17.312500 18380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.312553 18380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:17.312582 18380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:17.323228 18380 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:41971
I20260925 00:53:17.323266 18510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:41971 every 8 connection(s)
I20260925 00:53:17.324189 18380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:17.326333 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.313340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:57617 (local address 127.16.202.62:46271)
0925 00:53:17.313468 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.313471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.314721 (+ 1250us) server_negotiation.cc:407] Connection header received
0925 00:53:17.315747 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.315751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.315810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.315888 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.317601 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.318278 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.319481 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.319725 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.319817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.322641 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.322645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.322648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.322681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.324241 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.324888 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.324892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.324894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.325009 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.325355 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.325358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.325359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.325554 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.325629 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.326086 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.326203 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:53:17.327155 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.313959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.16.202.1:57617)
0925 00:53:17.314479 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.314523 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.315509 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.316137 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.316148 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.316620 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:17.317417 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.317440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.318407 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.318410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.319361 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.319371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.319891 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.319896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.320085 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.320708 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:17.320731 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:17.322434 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:17.324363 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.324370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.324386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.324757 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.325094 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.325097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.325099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.325262 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.325629 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:17.325634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:17.325873 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.326018 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.326275 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260925 00:53:17.327649 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18380
I20260925 00:53:17.327746 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:17.328040 17192 external_mini_cluster.cc:1428] Setting key cc092781d0843ed4232b07712f762caa
I20260925 00:53:17.328517 18511 heartbeater.cc:347] Connected to a master server at 127.16.202.62:46271
I20260925 00:53:17.328667 18511 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:17.329285 18511 heartbeater.cc:511] Master 127.16.202.62:46271 requested a full tablet report, sending...
I20260925 00:53:17.330790 18305 ts_manager.cc:194] Registered new tserver with Master: bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971)
I20260925 00:53:17.332331 18305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:57617
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:17.338939 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.330658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:56673 (local address 127.16.202.62:46271)
0925 00:53:17.330765 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.330769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.331003 (+ 234us) server_negotiation.cc:407] Connection header received
0925 00:53:17.331110 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.331113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.331157 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.331235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.331761 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.332272 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.334570 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.334787 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.334879 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.336123 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.336127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.336129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.336152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.337493 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.337932 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.337934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.337936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.338007 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.338216 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.338218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.338219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.338398 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.338471 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.338707 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.338799 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:53:17.338943 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.330591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.16.202.1:56673)
0925 00:53:17.330920 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.330935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.331033 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.331376 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.331380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.331550 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:17.331648 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.331653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.332394 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.332397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.333210 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.333221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.334925 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.334930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.335034 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.335525 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:17.335540 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:17.335977 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:17.337596 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.337600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.337602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.337843 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.338078 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.338080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.338081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.338156 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.338466 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:17.338468 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:17.338583 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.338668 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.338780 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:17.355612 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:46271
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:17.466274 18520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:17.466557 18520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:17.466602 18520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:17.470635 18520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:17.470690 18520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:17.470727 18520 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:17.470764 18520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:17.476125 18520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:17.477284 18520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:17.478327 18520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:17.487912 18525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:17.487922 18526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:17.487939 18528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:17.488121 18520 server_base.cc:1034] running on GCE node
I20260925 00:53:17.488582 18520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:17.489239 18520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:17.490396 18520 hybrid_clock.cc:648] HybridClock initialized: now 1790297597490376 us; error 38 us; skew 500 ppm
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:17.493080 18520 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:17.494017 18520 webserver.cc:460] Webserver started at http://127.16.202.2:45027/ using document root <none> and password file <none>
I20260925 00:53:17.494566 18520 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:17.494623 18520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:17.494812 18520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:17.496909 18520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "36e875bddedd412ea180f8bcbd7ffa9a"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "f27382926418019ee15485523b7c45d1"
server_key_iv: "75fd6d8680d7d4c9a7065f2db480d201"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.497447 18520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "36e875bddedd412ea180f8bcbd7ffa9a"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "f27382926418019ee15485523b7c45d1"
server_key_iv: "75fd6d8680d7d4c9a7065f2db480d201"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.501080 18520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:17.503369 18535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.504626 18520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:17.504745 18520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "36e875bddedd412ea180f8bcbd7ffa9a"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "f27382926418019ee15485523b7c45d1"
server_key_iv: "75fd6d8680d7d4c9a7065f2db480d201"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.504846 18520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:17.515313 18520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:17.518430 18520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:17.518607 18520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:17.520052 18520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:17.520112 18520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.520166 18520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:17.520196 18520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:17.530543 18520 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:36359
I20260925 00:53:17.530889 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:36359 every 8 connection(s)
I20260925 00:53:17.531497 18520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:17.531935 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18520
I20260925 00:53:17.532049 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:17.532310 17192 external_mini_cluster.cc:1428] Setting key d859a8b84e322bb4cb7eaf7811566ffb
I20260925 00:53:17.533767 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.520956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:57759 (local address 127.16.202.62:46271)
0925 00:53:17.521075 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.521078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.522276 (+ 1198us) server_negotiation.cc:407] Connection header received
0925 00:53:17.523416 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.523420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.523482 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.523554 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.524827 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.525367 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.526575 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.526827 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.526923 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.529944 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.529947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.529949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.529975 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.531435 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.532011 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.532014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.532016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.532109 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.532508 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.532512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.532514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.532775 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.532880 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.533500 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.533620 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:53:17.534744 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.521531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.16.202.2:57759)
0925 00:53:17.522132 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.522179 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.523161 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.523741 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.523748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.524087 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:17.524660 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.524703 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.525484 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.525488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.526460 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.526469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.526995 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.526999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.527159 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.527745 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:17.527766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:17.529750 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:17.531538 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.531543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.531558 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.531899 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.532215 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.532218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.532221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.532401 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.533038 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:17.533044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:17.533267 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.533438 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.533774 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":146,"threads_started":1}
I20260925 00:53:17.536078 18651 heartbeater.cc:347] Connected to a master server at 127.16.202.62:46271
I20260925 00:53:17.536222 18651 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:17.536760 18651 heartbeater.cc:511] Master 127.16.202.62:46271 requested a full tablet report, sending...
I20260925 00:53:17.537765 18308 ts_manager.cc:194] Registered new tserver with Master: 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359)
I20260925 00:53:17.538484 18308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:57759
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:17.545642 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.538366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:55425 (local address 127.16.202.62:46271)
0925 00:53:17.538504 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.538508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.538951 (+ 443us) server_negotiation.cc:407] Connection header received
0925 00:53:17.539054 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.539057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.539108 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.539199 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.539677 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.540455 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.541277 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.541611 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.541732 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.542688 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.542691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.542693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.542715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.544019 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.544361 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.544364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.544365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.544453 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.544741 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.544745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.544747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.545030 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.545128 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.545427 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.545522 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:53:17.545670 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.538500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.16.202.2:55425)
0925 00:53:17.538852 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.538866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.538969 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.539224 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.539227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.539446 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:17.539562 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.539569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.540593 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.540596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.541168 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.541177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.541722 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.541724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.541787 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.542259 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:17.542270 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:17.542568 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:17.544109 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.544112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.544113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.544287 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.544548 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.544551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.544552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.544628 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.545142 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:17.545145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:17.545245 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.545359 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.545495 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:17.560781 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:46271
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:17.675446 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:17.675731 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:17.675776 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:17.679991 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:17.680064 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:17.680105 18660 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:17.680137 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:17.685575 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:17.686751 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:17.687788 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:17.697427 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:17.697412 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:17.697541 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:17.697845 18660 server_base.cc:1034] running on GCE node
I20260925 00:53:17.698266 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:17.698936 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:17.700109 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790297597700075 us; error 44 us; skew 500 ppm
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:17.702777 18660 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:17.703665 18660 webserver.cc:460] Webserver started at http://127.16.202.3:42667/ using document root <none> and password file <none>
I20260925 00:53:17.704214 18660 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:17.704279 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:17.704463 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:17.706616 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "598949563bd8422e94571358c72f9852"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "0ee93a250b7520b1e30229107fa12be2"
server_key_iv: "24487955c629c00745befee5461ba9d5"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.707158 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "598949563bd8422e94571358c72f9852"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "0ee93a250b7520b1e30229107fa12be2"
server_key_iv: "24487955c629c00745befee5461ba9d5"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.710919 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:17.713325 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.714556 18660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 00:53:17.714665 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "598949563bd8422e94571358c72f9852"
format_stamp: "Formatted at 2026-09-25 00:53:17 on dist-test-slave-rpk1"
server_key: "0ee93a250b7520b1e30229107fa12be2"
server_key_iv: "24487955c629c00745befee5461ba9d5"
server_key_version: "encryptionkey@0"
I20260925 00:53:17.714762 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:17.724242 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:17.726913 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:17.727082 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:17.728487 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:17.728534 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:17.728585 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:17.728611 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:17.738111 18660 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:43857
I20260925 00:53:17.738198 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:43857 every 8 connection(s)
I20260925 00:53:17.738969 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:17.741326 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.729131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:58769 (local address 127.16.202.62:46271)
0925 00:53:17.729282 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.729287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.730560 (+ 1273us) server_negotiation.cc:407] Connection header received
0925 00:53:17.731792 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.731795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.731843 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.731910 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.733451 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.734029 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.735199 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.735428 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.735516 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.738027 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.738031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.738035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.738065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.739514 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.740019 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.740022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.740023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.740107 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.740357 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.740359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.740360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.740528 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.740596 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.741104 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.741207 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:17.742132 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.729938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.16.202.3:58769)
0925 00:53:17.730403 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.730458 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.731548 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.732101 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.732111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.732582 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:17.733286 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.733307 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.734154 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.734158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.735080 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.735090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.735561 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.735565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.735741 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.736255 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:17.736274 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:17.737819 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:17.739641 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.739646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.739660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.739926 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.740183 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.740185 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.740187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.740302 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.740588 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:17.740593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:17.740862 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.741035 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.741276 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260925 00:53:17.743280 18791 heartbeater.cc:347] Connected to a master server at 127.16.202.62:46271
I20260925 00:53:17.743456 18791 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:17.743916 18791 heartbeater.cc:511] Master 127.16.202.62:46271 requested a full tablet report, sending...
I20260925 00:53:17.744952 18305 ts_manager.cc:194] Registered new tserver with Master: 598949563bd8422e94571358c72f9852 (127.16.202.3:43857)
I20260925 00:53:17.745633 18305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:58769
I20260925 00:53:17.747200 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18660
I20260925 00:53:17.747288 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:17.747537 17192 external_mini_cluster.cc:1428] Setting key 24c3100f215f0a9bc928033a558b01c8
I20260925 00:53:17.749451 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:53:17.752298 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.745326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.16.202.3:56605)
0925 00:53:17.745700 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.745715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.745811 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.746037 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.746040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.746242 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:17.746338 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.746344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.747344 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.747347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.748122 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.748132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.748592 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.748595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.748707 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.749183 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:17.749194 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:17.749577 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:17.751101 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.751103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.751105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.751286 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.751528 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:17.751530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:17.751531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:17.751593 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:17.751931 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:17.751933 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:17.752000 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.752075 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.752172 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":127,"threads_started":1}
I20260925 00:53:17.752337 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.745428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:56605 (local address 127.16.202.62:46271)
0925 00:53:17.745546 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.745550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.745785 (+ 235us) server_negotiation.cc:407] Connection header received
0925 00:53:17.745893 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.745896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.745941 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.746012 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.746444 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.747203 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.748224 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.748484 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.748584 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.749685 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.749688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.749690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.749711 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.751000 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.751363 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.751366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.751368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.751455 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.751657 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.751660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.751661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.751852 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.751946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.752142 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.752229 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:53:17 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:17.774681 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.767072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.16.202.62:46271)
0925 00:53:17.767201 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.767205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.767216 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:53:17.767253 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.767256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.767293 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.767359 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:17.768065 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.768609 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.769483 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.769705 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.769792 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.771967 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:17.771970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:17.771972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:17.771998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:17.773262 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.773604 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.773606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.773608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.773674 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:17.773865 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:17.773867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:17.773868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:17.774036 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:17.774127 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.774388 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.774510 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:53:17.777801 18305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:17.780921 18305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:17.795389 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.789168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:41971 (local address 127.0.0.1:32888)
0925 00:53:17.789670 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.789720 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.789849 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.791205 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.791208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.791224 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:17.791305 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.791309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.792410 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.792414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.793994 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.794000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.794973 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.794976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.795068 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.795094 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.795145 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.795249 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":98,"thread_start_us":79,"threads_started":1}
I20260925 00:53:17.795397 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.790066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.16.202.3:43857)
0925 00:53:17.790435 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.790448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.790473 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:53:17.790548 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.790554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.790780 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.790955 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:17.791394 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.792534 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.794081 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.794716 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.794811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.794978 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.795142 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.795235 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":95,"threads_started":1}
I20260925 00:53:17.795790 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.789311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:36359 (local address 127.0.0.1:51708)
0925 00:53:17.790112 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.790124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.790415 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.791657 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.791660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.791678 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:17.791776 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.791782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.793088 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.793091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.794214 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.794225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.795393 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.795397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.795486 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.795509 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.795575 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.795640 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":93,"thread_start_us":112,"threads_started":1}
I20260925 00:53:17.795940 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.789331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.16.202.1:41971)
0925 00:53:17.790781 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.790796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.790817 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:53:17.790889 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.790894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.791073 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.791220 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:17.791428 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.792307 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.794113 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.794833 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.795523 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.795679 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.795728 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.795813 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":83,"threads_started":1}
I20260925 00:53:17.796201 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.790005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.16.202.2:36359)
0925 00:53:17.791234 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:17.791249 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:17.791269 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:53:17.791337 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:17.791342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:17.791516 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:17.791668 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:17.791889 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.792964 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.794337 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.795101 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.795189 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.795335 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:17.795977 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 00:53:17.796071 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":125,"threads_started":1}
I20260925 00:53:17.796249 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:17.789912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43857 (local address 127.0.0.1:33020)
0925 00:53:17.790167 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:17.790178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:17.790432 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:17.790952 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:17.790958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:17.790979 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:17.791106 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.791111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.792643 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.792647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.793958 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:17.793969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:17.794831 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:17.794835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:17.794961 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:17.795002 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:53:17.795104 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 00:53:17.796116 (+ 1012us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":6,"spinlock_wait_cycles":418176,"thread_start_us":81,"threads_started":1}
I20260925 00:53:17.798740 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 0714420c89e84542a4c158d4fbf810f6 (DEFAULT_TABLE table=test-table [id=6e2e8578032042e3ac6f9c86c6dc8dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:17.798862 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 0714420c89e84542a4c158d4fbf810f6 (DEFAULT_TABLE table=test-table [id=6e2e8578032042e3ac6f9c86c6dc8dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:17.798854 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 0714420c89e84542a4c158d4fbf810f6 (DEFAULT_TABLE table=test-table [id=6e2e8578032042e3ac6f9c86c6dc8dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:17.799695 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0714420c89e84542a4c158d4fbf810f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:17.799817 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0714420c89e84542a4c158d4fbf810f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:17.799822 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0714420c89e84542a4c158d4fbf810f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:17.806846 18814 tablet_bootstrap.cc:492] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Bootstrap starting.
I20260925 00:53:17.808488 18816 tablet_bootstrap.cc:492] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Bootstrap starting.
I20260925 00:53:17.809015 18815 tablet_bootstrap.cc:492] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Bootstrap starting.
I20260925 00:53:17.809249 18814 tablet_bootstrap.cc:654] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:17.809975 18814 log.cc:824] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:17.810711 18816 tablet_bootstrap.cc:654] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:17.810989 18815 tablet_bootstrap.cc:654] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:17.811563 18816 log.cc:824] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:17.811690 18815 log.cc:824] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:17.811930 18814 tablet_bootstrap.cc:492] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: No bootstrap required, opened a new log
I20260925 00:53:17.812119 18814 ts_tablet_manager.cc:1397] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:53:17.813547 18816 tablet_bootstrap.cc:492] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: No bootstrap required, opened a new log
I20260925 00:53:17.813673 18815 tablet_bootstrap.cc:492] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: No bootstrap required, opened a new log
I20260925 00:53:17.813719 18816 ts_tablet_manager.cc:1397] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:53:17.813848 18815 ts_tablet_manager.cc:1397] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:53:17.816128 18814 raft_consensus.cc:348] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:17.816404 18814 raft_consensus.cc:374] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:17.816454 18814 raft_consensus.cc:729] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf212653acac4518b7df4e302a4c8f7f, State: Initialized, Role: FOLLOWER
I20260925 00:53:17.816396 18816 raft_consensus.cc:348] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:17.816576 18816 raft_consensus.cc:374] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:17.816604 18816 raft_consensus.cc:729] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e875bddedd412ea180f8bcbd7ffa9a, State: Initialized, Role: FOLLOWER
I20260925 00:53:17.816931 18814 consensus_queue.cc:260] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:17.817039 18816 consensus_queue.cc:260] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:17.817610 18511 heartbeater.cc:503] Master 127.16.202.62:46271 was elected leader, sending a full tablet report...
I20260925 00:53:17.817502 18815 raft_consensus.cc:348] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:17.817714 18815 raft_consensus.cc:374] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:17.817763 18815 raft_consensus.cc:729] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598949563bd8422e94571358c72f9852, State: Initialized, Role: FOLLOWER
I20260925 00:53:17.817976 18816 ts_tablet_manager.cc:1428] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260925 00:53:17.818028 18814 ts_tablet_manager.cc:1428] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:53:17.818149 18815 consensus_queue.cc:260] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:17.818300 18651 heartbeater.cc:503] Master 127.16.202.62:46271 was elected leader, sending a full tablet report...
I20260925 00:53:17.819235 18791 heartbeater.cc:503] Master 127.16.202.62:46271 was elected leader, sending a full tablet report...
I20260925 00:53:17.820096 18815 ts_tablet_manager.cc:1428] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 00:53:17.825588 18512 tablet.cc:2369] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:17.989890 18792 tablet.cc:2369] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:18.033102 18652 tablet.cc:2369] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:18.036586 18822 raft_consensus.cc:482] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:18.036809 18822 raft_consensus.cc:504] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.038013 18822 leader_election.cc:288] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971), 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359)
I20260925 00:53:18.041733 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.038617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:45797 (local address 127.16.202.2:36359)
0925 00:53:18.038744 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.038749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.038762 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:18.038809 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.038811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.038845 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.038908 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:18.039140 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.039657 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.040713 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.041247 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.041335 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.041374 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.041572 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.041627 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:53:18.041788 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.038249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:36359 (local address 127.16.202.3:45797)
0925 00:53:18.038611 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.038625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.038720 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.038923 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.038925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.038937 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:18.039039 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.039043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.039761 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.039764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.040567 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.040575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.041341 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.041345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.041460 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.041486 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.041561 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.041631 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":30,"thread_start_us":122,"threads_started":1}
I20260925 00:53:18.042172 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.038234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:41971 (local address 127.16.202.3:41629)
0925 00:53:18.038354 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.038369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.038473 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.038992 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.038994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.039009 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:18.039092 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.039098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.039768 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.039770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.041033 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.041043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.041774 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.041777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.041881 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.041909 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.041974 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.042047 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260925 00:53:18.042255 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.038322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:41629 (local address 127.16.202.1:41971)
0925 00:53:18.038490 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.038495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.038528 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:53:18.038578 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.038580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.038616 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.038676 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:18.039189 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.039671 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.041149 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.041666 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.041770 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.041814 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.042071 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.042142 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:53:18.042407 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0714420c89e84542a4c158d4fbf810f6" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" is_pre_election: true
I20260925 00:53:18.042691 18605 raft_consensus.cc:2454] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 0.
I20260925 00:53:18.042738 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0714420c89e84542a4c158d4fbf810f6" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf212653acac4518b7df4e302a4c8f7f" is_pre_election: true
I20260925 00:53:18.042965 18465 raft_consensus.cc:2454] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 0.
I20260925 00:53:18.043144 18678 leader_election.cc:302] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e875bddedd412ea180f8bcbd7ffa9a, 598949563bd8422e94571358c72f9852; no voters:
I20260925 00:53:18.043430 18822 raft_consensus.cc:2781] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:18.043540 18822 raft_consensus.cc:482] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:18.043593 18822 raft_consensus.cc:3037] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.045082 18822 raft_consensus.cc:504] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.045414 18822 leader_election.cc:288] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 election: Requested vote from peers bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971), 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359)
I20260925 00:53:18.045783 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0714420c89e84542a4c158d4fbf810f6" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf212653acac4518b7df4e302a4c8f7f"
I20260925 00:53:18.045944 18465 raft_consensus.cc:3037] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.046989 18465 raft_consensus.cc:2454] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 1.
I20260925 00:53:18.047273 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0714420c89e84542a4c158d4fbf810f6" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e875bddedd412ea180f8bcbd7ffa9a"
I20260925 00:53:18.047361 18677 leader_election.cc:302] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598949563bd8422e94571358c72f9852, bf212653acac4518b7df4e302a4c8f7f; no voters:
I20260925 00:53:18.047406 18605 raft_consensus.cc:3037] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.047529 18822 raft_consensus.cc:2781] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:18.047756 18822 raft_consensus.cc:686] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 1 LEADER]: Becoming Leader. State: Replica: 598949563bd8422e94571358c72f9852, State: Running, Role: LEADER
I20260925 00:53:18.048115 18822 consensus_queue.cc:237] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.048666 18605 raft_consensus.cc:2454] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 1.
I20260925 00:53:18.051585 18308 catalog_manager.cc:5495] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 598949563bd8422e94571358c72f9852 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "598949563bd8422e94571358c72f9852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } health_report { overall_health: HEALTHY } } }
I20260925 00:53:18.095577 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.091573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.16.202.3:43857)
0925 00:53:18.091739 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.091743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.091760 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:53:18.092039 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.092042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.092101 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.092197 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:18.092484 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.093069 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.093956 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.094181 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.094312 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.094693 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.095159 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.095298 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.095399 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.095458 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 00:53:18.101518 18465 raft_consensus.cc:1261] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Refusing update from remote peer 598949563bd8422e94571358c72f9852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:18.101516 18605 raft_consensus.cc:1261] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Refusing update from remote peer 598949563bd8422e94571358c72f9852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:18.102241 18824 consensus_queue.cc:1035] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:18.102427 18822 consensus_queue.cc:1035] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:18.124965 18305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:18.125147 18305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:18.127665 18305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: Sending DeleteTablet for 3 replicas of tablet 0714420c89e84542a4c158d4fbf810f6
I20260925 00:53:18.128705 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 0714420c89e84542a4c158d4fbf810f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:51708
I20260925 00:53:18.129014 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 0714420c89e84542a4c158d4fbf810f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33020
I20260925 00:53:18.129454 18837 tablet_replica.cc:330] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: stopping tablet replica
I20260925 00:53:18.129376 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 0714420c89e84542a4c158d4fbf810f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:32888
I20260925 00:53:18.129519 18308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:53:18.129839 18839 tablet_replica.cc:330] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: stopping tablet replica
I20260925 00:53:18.129926 18837 raft_consensus.cc:2229] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 00:53:18.129963 18308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:18.130106 18839 raft_consensus.cc:2229] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:18.130250 18837 raft_consensus.cc:2258] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:18.130326 18839 raft_consensus.cc:2258] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:18.131745 18839 ts_tablet_manager.cc:1905] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:18.131745 18837 ts_tablet_manager.cc:1905] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:18.134408 18839 ts_tablet_manager.cc:1918] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:18.134402 18837 ts_tablet_manager.cc:1918] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:18.134502 18837 log.cc:1196] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/0714420c89e84542a4c158d4fbf810f6
I20260925 00:53:18.134531 18839 log.cc:1196] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/0714420c89e84542a4c158d4fbf810f6
I20260925 00:53:18.134812 18837 ts_tablet_manager.cc:1939] T 0714420c89e84542a4c158d4fbf810f6 P 36e875bddedd412ea180f8bcbd7ffa9a: Deleting consensus metadata
I20260925 00:53:18.134812 18839 ts_tablet_manager.cc:1939] T 0714420c89e84542a4c158d4fbf810f6 P bf212653acac4518b7df4e302a4c8f7f: Deleting consensus metadata
I20260925 00:53:18.136211 18294 catalog_manager.cc:4841] TS 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359): tablet 0714420c89e84542a4c158d4fbf810f6 (table test-table [id=6e2e8578032042e3ac6f9c86c6dc8dfa]) successfully deleted
I20260925 00:53:18.136255 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 7c97bf9439f54bab98641d5a222b05a9 (DEFAULT_TABLE table=test-table [id=138c7fd980534418b56322691a4d37c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:18.136494 18293 catalog_manager.cc:4841] TS bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971): tablet 0714420c89e84542a4c158d4fbf810f6 (table test-table [id=6e2e8578032042e3ac6f9c86c6dc8dfa]) successfully deleted
I20260925 00:53:18.136521 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c97bf9439f54bab98641d5a222b05a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:18.136565 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 7c97bf9439f54bab98641d5a222b05a9 (DEFAULT_TABLE table=test-table [id=138c7fd980534418b56322691a4d37c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:18.136833 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c97bf9439f54bab98641d5a222b05a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:18.138690 18814 tablet_bootstrap.cc:492] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Bootstrap starting.
I20260925 00:53:18.138887 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 7c97bf9439f54bab98641d5a222b05a9 (DEFAULT_TABLE table=test-table [id=138c7fd980534418b56322691a4d37c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:18.139130 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c97bf9439f54bab98641d5a222b05a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:18.139633 18838 tablet_replica.cc:330] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: stopping tablet replica
I20260925 00:53:18.139648 18814 tablet_bootstrap.cc:654] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:18.139972 18838 raft_consensus.cc:2229] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:18.140446 18838 raft_consensus.cc:2258] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:18.140802 18815 tablet_bootstrap.cc:492] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Bootstrap starting.
I20260925 00:53:18.141446 18816 tablet_bootstrap.cc:492] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Bootstrap starting.
I20260925 00:53:18.141717 18815 tablet_bootstrap.cc:654] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:18.142165 18814 tablet_bootstrap.cc:492] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: No bootstrap required, opened a new log
I20260925 00:53:18.142254 18814 ts_tablet_manager.cc:1397] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:53:18.142314 18816 tablet_bootstrap.cc:654] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:18.142978 18814 raft_consensus.cc:348] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.143096 18814 raft_consensus.cc:374] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:18.143126 18814 raft_consensus.cc:729] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf212653acac4518b7df4e302a4c8f7f, State: Initialized, Role: FOLLOWER
I20260925 00:53:18.143246 18814 consensus_queue.cc:260] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.143486 18815 tablet_bootstrap.cc:492] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: No bootstrap required, opened a new log
I20260925 00:53:18.143504 18814 ts_tablet_manager.cc:1428] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:53:18.143561 18815 ts_tablet_manager.cc:1397] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 00:53:18.143569 18816 tablet_bootstrap.cc:492] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: No bootstrap required, opened a new log
I20260925 00:53:18.143630 18816 ts_tablet_manager.cc:1397] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:18.144001 18815 raft_consensus.cc:348] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.144114 18815 raft_consensus.cc:374] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:18.144145 18815 raft_consensus.cc:729] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598949563bd8422e94571358c72f9852, State: Initialized, Role: FOLLOWER
I20260925 00:53:18.144254 18815 consensus_queue.cc:260] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.144376 18816 raft_consensus.cc:348] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.144469 18816 raft_consensus.cc:374] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:18.144498 18816 raft_consensus.cc:729] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e875bddedd412ea180f8bcbd7ffa9a, State: Initialized, Role: FOLLOWER
I20260925 00:53:18.144529 18815 ts_tablet_manager.cc:1428] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:18.144651 18824 raft_consensus.cc:482] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:18.144630 18816 consensus_queue.cc:260] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.144754 18824 raft_consensus.cc:504] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.144932 18816 ts_tablet_manager.cc:1428] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:53:18.145139 18824 leader_election.cc:288] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971), 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359)
I20260925 00:53:18.145421 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c97bf9439f54bab98641d5a222b05a9" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf212653acac4518b7df4e302a4c8f7f" is_pre_election: true
I20260925 00:53:18.145545 18465 raft_consensus.cc:2454] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 0.
I20260925 00:53:18.145846 18677 leader_election.cc:302] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598949563bd8422e94571358c72f9852, bf212653acac4518b7df4e302a4c8f7f; no voters:
I20260925 00:53:18.145916 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c97bf9439f54bab98641d5a222b05a9" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" is_pre_election: true
I20260925 00:53:18.145998 18824 raft_consensus.cc:2781] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:18.146025 18605 raft_consensus.cc:2454] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 0.
I20260925 00:53:18.146064 18824 raft_consensus.cc:482] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:18.146106 18824 raft_consensus.cc:3037] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.148298 18824 raft_consensus.cc:504] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.148530 18838 ts_tablet_manager.cc:1905] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:18.148700 18824 leader_election.cc:288] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 election: Requested vote from peers bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971), 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359)
I20260925 00:53:18.149040 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c97bf9439f54bab98641d5a222b05a9" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf212653acac4518b7df4e302a4c8f7f"
I20260925 00:53:18.149039 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c97bf9439f54bab98641d5a222b05a9" candidate_uuid: "598949563bd8422e94571358c72f9852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e875bddedd412ea180f8bcbd7ffa9a"
I20260925 00:53:18.149165 18465 raft_consensus.cc:3037] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.149165 18605 raft_consensus.cc:3037] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.150269 18605 raft_consensus.cc:2454] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 1.
I20260925 00:53:18.150280 18465 raft_consensus.cc:2454] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598949563bd8422e94571358c72f9852 in term 1.
I20260925 00:53:18.150581 18678 leader_election.cc:302] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36e875bddedd412ea180f8bcbd7ffa9a, 598949563bd8422e94571358c72f9852, bf212653acac4518b7df4e302a4c8f7f; no voters:
I20260925 00:53:18.151044 18824 raft_consensus.cc:2781] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:18.151129 18824 raft_consensus.cc:686] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 1 LEADER]: Becoming Leader. State: Replica: 598949563bd8422e94571358c72f9852, State: Running, Role: LEADER
I20260925 00:53:18.151260 18824 consensus_queue.cc:237] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.152952 18305 catalog_manager.cc:5495] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 598949563bd8422e94571358c72f9852 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "598949563bd8422e94571358c72f9852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } health_report { overall_health: HEALTHY } } }
I20260925 00:53:18.155776 18838 ts_tablet_manager.cc:1918] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:18.155892 18838 log.cc:1196] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/0714420c89e84542a4c158d4fbf810f6
I20260925 00:53:18.156188 18838 ts_tablet_manager.cc:1939] T 0714420c89e84542a4c158d4fbf810f6 P 598949563bd8422e94571358c72f9852: Deleting consensus metadata
I20260925 00:53:18.157518 18293 catalog_manager.cc:4841] TS 598949563bd8422e94571358c72f9852 (127.16.202.3:43857): tablet 0714420c89e84542a4c158d4fbf810f6 (table test-table [id=6e2e8578032042e3ac6f9c86c6dc8dfa]) successfully deleted
Sep 25 00:53:18 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297596, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:18.195698 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.186888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.16.202.62:46271)
0925 00:53:18.187019 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.187024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.187222 (+ 198us) server_negotiation.cc:407] Connection header received
0925 00:53:18.187331 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.187333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.187387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.187486 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:18.188145 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.189042 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.189804 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.190136 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.190229 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.192547 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:18.192550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:18.192552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:18.192572 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:18.193854 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:18.194389 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:18.194392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:18.194393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:18.194471 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:18.194756 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:18.194758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:18.194760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:18.194953 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:18.195033 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.195376 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.195553 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":39}
I20260925 00:53:18.195704 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.186837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46271 (local address 127.0.0.1:55424)
0925 00:53:18.187139 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.187155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.187260 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.187518 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.187520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.187930 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:18.188036 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.188042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.189138 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.189141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.189701 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.189709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.190256 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.190259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.190324 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.190909 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:18.190921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:18.192416 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:18.193957 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:18.193968 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:18.193976 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:18.194274 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:18.194555 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:18.194557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:18.194559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:18.194646 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:18.195048 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:18.195055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:18.195187 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.195305 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.195480 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
I20260925 00:53:18.199842 18305 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:55424:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 00:53:18.200722 18305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:18.205926 18725 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6f134bb3147f8988f1b08a6118a64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f6aef4e2287448ea6eead5561f0a02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:18.205926 18585 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6f134bb3147f8988f1b08a6118a64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f6aef4e2287448ea6eead5561f0a02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:18.205926 18445 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6f134bb3147f8988f1b08a6118a64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f6aef4e2287448ea6eead5561f0a02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:18.206357 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb6f134bb3147f8988f1b08a6118a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:18.206357 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb6f134bb3147f8988f1b08a6118a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:18.206364 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb6f134bb3147f8988f1b08a6118a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:18.209349 18816 tablet_bootstrap.cc:492] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: Bootstrap starting.
I20260925 00:53:18.209388 18814 tablet_bootstrap.cc:492] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: Bootstrap starting.
I20260925 00:53:18.209523 18815 tablet_bootstrap.cc:492] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: Bootstrap starting.
I20260925 00:53:18.210386 18815 tablet_bootstrap.cc:654] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:18.210386 18816 tablet_bootstrap.cc:654] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:18.210400 18814 tablet_bootstrap.cc:654] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:18.211957 18815 tablet_bootstrap.cc:492] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: No bootstrap required, opened a new log
I20260925 00:53:18.212046 18815 ts_tablet_manager.cc:1397] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 00:53:18.212497 18815 raft_consensus.cc:348] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.212601 18815 raft_consensus.cc:374] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:18.212631 18815 raft_consensus.cc:729] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598949563bd8422e94571358c72f9852, State: Initialized, Role: FOLLOWER
I20260925 00:53:18.212792 18815 consensus_queue.cc:260] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.213085 18815 ts_tablet_manager.cc:1428] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:18.213042 18824 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.213169 18824 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.213981 18816 tablet_bootstrap.cc:492] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: No bootstrap required, opened a new log
I20260925 00:53:18.214105 18816 ts_tablet_manager.cc:1397] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 00:53:18.214566 18816 raft_consensus.cc:348] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.214689 18816 raft_consensus.cc:374] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:18.214723 18816 raft_consensus.cc:729] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e875bddedd412ea180f8bcbd7ffa9a, State: Initialized, Role: FOLLOWER
I20260925 00:53:18.214852 18816 consensus_queue.cc:260] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.215097 18821 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.215248 18821 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.215580 18814 tablet_bootstrap.cc:492] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: No bootstrap required, opened a new log
I20260925 00:53:18.215673 18814 ts_tablet_manager.cc:1397] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 00:53:18.216135 18814 raft_consensus.cc:348] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.216274 18814 raft_consensus.cc:374] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:18.216308 18814 raft_consensus.cc:729] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf212653acac4518b7df4e302a4c8f7f, State: Initialized, Role: FOLLOWER
I20260925 00:53:18.216465 18814 consensus_queue.cc:260] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.216794 18814 ts_tablet_manager.cc:1428] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:53:18.216799 18820 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.216987 18820 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.215107 18816 ts_tablet_manager.cc:1428] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:53:18.336525 18821 raft_consensus.cc:482] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:18.336690 18821 raft_consensus.cc:504] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.337880 18821 leader_election.cc:288] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971), 598949563bd8422e94571358c72f9852 (127.16.202.3:43857)
I20260925 00:53:18.341703 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.338201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:59565 (local address 127.16.202.1:41971)
0925 00:53:18.338337 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.338341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.338608 (+ 267us) server_negotiation.cc:407] Connection header received
0925 00:53:18.338748 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.338751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.338820 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.338896 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:18.339190 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.339687 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.340774 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.341160 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.341266 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.341321 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.341524 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.341599 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:53:18.341727 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.338058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:41971 (local address 127.16.202.2:59565)
0925 00:53:18.338368 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.338384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.338631 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.338958 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.338961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.338981 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:18.339082 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.339087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.339788 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.339792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.340617 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.340626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.341274 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.341278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.341409 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.341429 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.341493 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.341560 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":39,"thread_start_us":98,"threads_started":1}
I20260925 00:53:18.342260 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb6f134bb3147f8988f1b08a6118a64" candidate_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf212653acac4518b7df4e302a4c8f7f" is_pre_election: true
I20260925 00:53:18.342422 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.338688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43857 (local address 127.16.202.2:42709)
0925 00:53:18.339030 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.339042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.339129 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.339351 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.339354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.339368 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:18.339876 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.339883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.340574 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.340577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.341390 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.341398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.342086 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.342090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.342183 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.342203 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.342261 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.342309 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260925 00:53:18.342430 18465 raft_consensus.cc:2454] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e875bddedd412ea180f8bcbd7ffa9a in term 0.
I20260925 00:53:18.342473 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.338809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:42709 (local address 127.16.202.3:43857)
0925 00:53:18.338925 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.338930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.339113 (+ 183us) server_negotiation.cc:407] Connection header received
0925 00:53:18.339210 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.339213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.339261 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.339327 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:18.339990 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.340474 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.341504 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.341972 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.342088 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.342128 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.342289 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.342361 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:53:18.342816 18537 leader_election.cc:302] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e875bddedd412ea180f8bcbd7ffa9a, bf212653acac4518b7df4e302a4c8f7f; no voters:
I20260925 00:53:18.342921 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb6f134bb3147f8988f1b08a6118a64" candidate_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598949563bd8422e94571358c72f9852" is_pre_election: true
I20260925 00:53:18.343081 18821 raft_consensus.cc:2781] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:18.343137 18821 raft_consensus.cc:482] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:18.343178 18821 raft_consensus.cc:3037] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.343156 18745 raft_consensus.cc:2454] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e875bddedd412ea180f8bcbd7ffa9a in term 0.
I20260925 00:53:18.344188 18821 raft_consensus.cc:504] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.344512 18821 leader_election.cc:288] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [CANDIDATE]: Term 1 election: Requested vote from peers bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971), 598949563bd8422e94571358c72f9852 (127.16.202.3:43857)
I20260925 00:53:18.344863 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb6f134bb3147f8988f1b08a6118a64" candidate_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf212653acac4518b7df4e302a4c8f7f"
I20260925 00:53:18.344937 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb6f134bb3147f8988f1b08a6118a64" candidate_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598949563bd8422e94571358c72f9852"
I20260925 00:53:18.344986 18465 raft_consensus.cc:3037] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.345057 18745 raft_consensus.cc:3037] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:18.345839 18465 raft_consensus.cc:2454] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e875bddedd412ea180f8bcbd7ffa9a in term 1.
I20260925 00:53:18.345988 18745 raft_consensus.cc:2454] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e875bddedd412ea180f8bcbd7ffa9a in term 1.
I20260925 00:53:18.346094 18537 leader_election.cc:302] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e875bddedd412ea180f8bcbd7ffa9a, bf212653acac4518b7df4e302a4c8f7f; no voters:
I20260925 00:53:18.346257 18821 raft_consensus.cc:2781] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:18.346520 18821 raft_consensus.cc:686] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 LEADER]: Becoming Leader. State: Replica: 36e875bddedd412ea180f8bcbd7ffa9a, State: Running, Role: LEADER
I20260925 00:53:18.346751 18821 consensus_queue.cc:237] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } }
I20260925 00:53:18.347896 18855 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: TxnStatusTablet state changed. Reason: New leader 36e875bddedd412ea180f8bcbd7ffa9a. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.348011 18855 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:18.348263 18857 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 00:53:18.348443 18857 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 00:53:18.349143 18305 catalog_manager.cc:5495] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2). New cstate: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:18.377429 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.373148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:36359 (local address 127.0.0.1:51712)
0925 00:53:18.373354 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.373376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.373533 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.374230 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.374233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.374242 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:18.374394 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.374399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.375119 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.375123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.375791 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.375802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.376217 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.376220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.376313 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.376362 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.377120 (+ 758us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.377138 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.377203 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.377307 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 00:53:18.377430 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.373829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.16.202.2:36359)
0925 00:53:18.373966 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.373970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.373983 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:18.374052 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.374054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.374104 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.374193 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:18.374505 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.375008 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.375899 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.376115 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.376200 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.376500 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.377035 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.377138 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.377214 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.377266 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:53:18.380640 18465 raft_consensus.cc:1261] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Refusing update from remote peer 36e875bddedd412ea180f8bcbd7ffa9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:18.380882 18745 raft_consensus.cc:1261] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Refusing update from remote peer 36e875bddedd412ea180f8bcbd7ffa9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:18.381155 18821 consensus_queue.cc:1035] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:18.381366 18855 consensus_queue.cc:1035] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:18.382807 18820 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: TxnStatusTablet state changed. Reason: New leader 36e875bddedd412ea180f8bcbd7ffa9a. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.382925 18820 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.383033 18824 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: TxnStatusTablet state changed. Reason: New leader 36e875bddedd412ea180f8bcbd7ffa9a. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.383127 18824 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.385106 18828 mvcc.cc:205] Tried to move back new op lower bound from 7333058962962124800 to 7333058962830610432. Current Snapshot: MvccSnapshot[applied={T|T < 7333058962962124800}]
I20260925 00:53:18.385349 18824 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.385460 18824 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 598949563bd8422e94571358c72f9852: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.385488 18820 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.385578 18820 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P bf212653acac4518b7df4e302a4c8f7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:18.385665 18827 mvcc.cc:205] Tried to move back new op lower bound from 7333058962962124800 to 7333058962830610432. Current Snapshot: MvccSnapshot[applied={T|T < 7333058962962124800}]
I20260925 00:53:18.388218 18862 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.388252 18855 tablet_replica.cc:439] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 } } peers { permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 } } peers { permanent_uuid: "598949563bd8422e94571358c72f9852" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43857 } } }
I20260925 00:53:18.388401 18855 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:18.388541 18862 tablet_replica.cc:442] T dfb6f134bb3147f8988f1b08a6118a64 P 36e875bddedd412ea180f8bcbd7ffa9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:18.399832 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.396608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:36359 (local address 127.16.202.3:57453)
0925 00:53:18.396911 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.396926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.397038 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.397251 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.397254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.397318 (+ 64us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:18.397408 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.397413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.398115 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.398119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.398741 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.398750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.399170 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.399174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.399252 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.399302 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.399558 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.399564 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.399622 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.399708 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
I20260925 00:53:18.399823 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.396819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:57453 (local address 127.16.202.2:36359)
0925 00:53:18.396929 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.396933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.396992 (+ 59us) server_negotiation.cc:407] Connection header received
0925 00:53:18.397128 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.397131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.397178 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.397237 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:18.397503 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.398015 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.398847 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.399075 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.399158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.399419 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.399492 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.399547 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.399631 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.399675 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:18.406432 18465 raft_consensus.cc:1261] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Refusing update from remote peer 598949563bd8422e94571358c72f9852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:18.406456 18605 raft_consensus.cc:1261] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Refusing update from remote peer 598949563bd8422e94571358c72f9852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:18.406898 18824 consensus_queue.cc:1035] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf212653acac4518b7df4e302a4c8f7f" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:18.407056 18832 consensus_queue.cc:1035] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e875bddedd412ea180f8bcbd7ffa9a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:18.426807 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.423037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:37175 (local address 127.16.202.3:43857)
0925 00:53:18.423154 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.423158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.423348 (+ 190us) server_negotiation.cc:407] Connection header received
0925 00:53:18.423517 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.423520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.423571 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.423654 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:18.423891 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.424506 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.425598 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.425886 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.425991 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.426318 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.426406 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.426479 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.426601 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.426666 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:53:18.428476 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.422945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43857 (local address 127.16.202.2:37175)
0925 00:53:18.423280 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:18.423295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:18.423408 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:18.423652 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:18.423655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:18.423663 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:18.423782 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.423789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.424628 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.424631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.425496 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:18.425505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.426002 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.426006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:18.426113 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.426178 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:18.426489 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:18.426494 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:53:18.428218 (+ 1724us) client_negotiation.cc:240] Negotiation successful
0925 00:53:18.428307 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":108,"threads_started":1}
I20260925 00:53:18.447712 18305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:18.447917 18305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:18.450129 18305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d3f74b54b2146529807076317568c27: Sending DeleteTablet for 3 replicas of tablet 7c97bf9439f54bab98641d5a222b05a9
I20260925 00:53:18.450793 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c97bf9439f54bab98641d5a222b05a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:32888
I20260925 00:53:18.450835 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c97bf9439f54bab98641d5a222b05a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:51708
I20260925 00:53:18.450989 18839 tablet_replica.cc:330] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: stopping tablet replica
I20260925 00:53:18.451035 18837 tablet_replica.cc:330] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: stopping tablet replica
I20260925 00:53:18.451112 18839 raft_consensus.cc:2229] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:18.451125 18837 raft_consensus.cc:2229] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:18.451234 18839 raft_consensus.cc:2258] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:18.451234 18837 raft_consensus.cc:2258] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:18.451690 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c97bf9439f54bab98641d5a222b05a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33020
I20260925 00:53:18.451851 18838 tablet_replica.cc:330] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: stopping tablet replica
I20260925 00:53:18.451951 18838 raft_consensus.cc:2229] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:18.452105 18837 ts_tablet_manager.cc:1905] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:18.452136 18838 raft_consensus.cc:2258] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:18.452827 18838 ts_tablet_manager.cc:1905] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:18.454960 18837 ts_tablet_manager.cc:1918] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:18.455027 18837 log.cc:1196] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/7c97bf9439f54bab98641d5a222b05a9
I20260925 00:53:18.455251 18837 ts_tablet_manager.cc:1939] T 7c97bf9439f54bab98641d5a222b05a9 P 36e875bddedd412ea180f8bcbd7ffa9a: Deleting consensus metadata
I20260925 00:53:18.455457 18838 ts_tablet_manager.cc:1918] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:18.455516 18838 log.cc:1196] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/7c97bf9439f54bab98641d5a222b05a9
I20260925 00:53:18.455718 18838 ts_tablet_manager.cc:1939] T 7c97bf9439f54bab98641d5a222b05a9 P 598949563bd8422e94571358c72f9852: Deleting consensus metadata
I20260925 00:53:18.456036 18294 catalog_manager.cc:4841] TS 36e875bddedd412ea180f8bcbd7ffa9a (127.16.202.2:36359): tablet 7c97bf9439f54bab98641d5a222b05a9 (table test-table [id=138c7fd980534418b56322691a4d37c0]) successfully deleted
I20260925 00:53:18.456344 18293 catalog_manager.cc:4841] TS 598949563bd8422e94571358c72f9852 (127.16.202.3:43857): tablet 7c97bf9439f54bab98641d5a222b05a9 (table test-table [id=138c7fd980534418b56322691a4d37c0]) successfully deleted
I20260925 00:53:18.456615 18839 ts_tablet_manager.cc:1905] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:18.459030 18839 ts_tablet_manager.cc:1918] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:18.459101 18839 log.cc:1196] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/7c97bf9439f54bab98641d5a222b05a9
I20260925 00:53:18.459287 18839 ts_tablet_manager.cc:1939] T 7c97bf9439f54bab98641d5a222b05a9 P bf212653acac4518b7df4e302a4c8f7f: Deleting consensus metadata
I20260925 00:53:18.460165 18293 catalog_manager.cc:4841] TS bf212653acac4518b7df4e302a4c8f7f (127.16.202.1:41971): tablet 7c97bf9439f54bab98641d5a222b05a9 (table test-table [id=138c7fd980534418b56322691a4d37c0]) successfully deleted
Sep 25 00:53:18 dist-test-slave-rpk1 krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:18.467252 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.452758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.16.202.1:41971)
0925 00:53:18.452941 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.452946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.456504 (+ 3558us) server_negotiation.cc:407] Connection header received
0925 00:53:18.456757 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.456761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.456819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.456905 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:18.457753 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.458455 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.459633 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.459942 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.460047 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.463700 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:18.463709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:18.463721 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:18.463745 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:18.465392 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:18.465854 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:18.465860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:18.465861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:18.465962 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:18.466214 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:18.466216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:18.466217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:18.466469 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:18.466624 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.466822 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.466933 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260925 00:53:18.467764 18504 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32892
I20260925 00:53:18.477579 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:18.469607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.16.202.62:46271)
0925 00:53:18.469739 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:18.469744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:18.469813 (+ 69us) server_negotiation.cc:407] Connection header received
0925 00:53:18.469923 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:18.469926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:18.469975 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:18.470052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:18.470688 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.471493 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.472370 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:18.472708 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:18.472809 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:18.474173 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:18.474177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:18.474180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:18.474210 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:18.475717 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:18.476151 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:18.476155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:18.476156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:18.476244 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:18.476573 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:18.476577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:18.476578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:18.476895 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:18.477025 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:18.477254 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 00:53:18.477375 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 00:53:18.478094 18305 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55426
I20260925 00:53:18.479387 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18380
I20260925 00:53:18.485090 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18520
I20260925 00:53:18.490917 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18660
I20260925 00:53:18.496260 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18276
2026-09-25T00:53:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2721 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:18 dist-test-slave-rpk1 krb5kdc[18877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:18 dist-test-slave-rpk1 krb5kdc[18877](info): set up 2 sockets
Sep 25 00:53:18 dist-test-slave-rpk1 krb5kdc[18877](info): commencing operation
krb5kdc: starting...
W20260925 00:53:19.456467 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:19 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:19.577780 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:46661
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:46661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:19.693934 18890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:19.694237 18890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:19.694280 18890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:19.698551 18890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:19.698632 18890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 00:53:19.698648 18890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:19.698666 18890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:19.698681 18890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:19.698696 18890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:19.704263 18890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:46661
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:19.705466 18890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:19.706542 18890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:19.715296 18895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:19.715296 18898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:19.715296 18896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:19.715358 18890 server_base.cc:1034] running on GCE node
I20260925 00:53:19.716107 18890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:19.717325 18890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:19.718514 18890 hybrid_clock.cc:648] HybridClock initialized: now 1790297599718500 us; error 30 us; skew 500 ppm
Sep 25 00:53:19 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297599, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:19.721401 18890 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:19.722369 18890 webserver.cc:460] Webserver started at http://127.16.202.62:36167/ using document root <none> and password file <none>
I20260925 00:53:19.722918 18890 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:19.722978 18890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:19.723173 18890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:19.725420 18890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "5d982a314a894083a902fc3cbaf3da8d"
format_stamp: "Formatted at 2026-09-25 00:53:19 on dist-test-slave-rpk1"
server_key: "e13971554a30cb0a473751d629f616ab"
server_key_iv: "b93ec74a9a96fff787378f33134b0e7d"
server_key_version: "encryptionkey@0"
I20260925 00:53:19.726025 18890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "5d982a314a894083a902fc3cbaf3da8d"
format_stamp: "Formatted at 2026-09-25 00:53:19 on dist-test-slave-rpk1"
server_key: "e13971554a30cb0a473751d629f616ab"
server_key_iv: "b93ec74a9a96fff787378f33134b0e7d"
server_key_version: "encryptionkey@0"
I20260925 00:53:19.730028 18890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 00:53:19.732733 18905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:19.734108 18890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:53:19.734242 18890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "5d982a314a894083a902fc3cbaf3da8d"
format_stamp: "Formatted at 2026-09-25 00:53:19 on dist-test-slave-rpk1"
server_key: "e13971554a30cb0a473751d629f616ab"
server_key_iv: "b93ec74a9a96fff787378f33134b0e7d"
server_key_version: "encryptionkey@0"
I20260925 00:53:19.734362 18890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:19.779405 18890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:19.782567 18890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:19.782761 18890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:19.790501 18890 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:46661
I20260925 00:53:19.790510 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:46661 every 8 connection(s)
I20260925 00:53:19.791453 18890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:19.794281 18968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:19.794852 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18890
I20260925 00:53:19.794963 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:19.795235 17192 external_mini_cluster.cc:1428] Setting key cb135b7f601ae1206d1d7bfc03dc3c81
I20260925 00:53:19.801388 18968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Bootstrap starting.
I20260925 00:53:19.804107 18968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Neither blocks nor log segments found. Creating new log.
Sep 25 00:53:19 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:19.804926 18968 log.cc:824] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:19.807104 18968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: No bootstrap required, opened a new log
I20260925 00:53:19.810088 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:19.796610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.16.202.62:46661)
0925 00:53:19.797046 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:19.797061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:19.797098 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:53:19.797763 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:19.797782 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:19.798137 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:19.798423 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:19.799322 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:19.800331 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:19.801469 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:19.801996 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:19.802113 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:19.804876 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:19.804882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:19.804900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:19.804947 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:19.807207 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:19.807721 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:19.807727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:19.807731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:19.807861 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:19.808144 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:19.808146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:19.808148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:19.808563 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:19.808779 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:19.809027 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 00:53:19.809312 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":111,"threads_started":1}
I20260925 00:53:19.810655 18968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d982a314a894083a902fc3cbaf3da8d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46661 } }
I20260925 00:53:19.810899 18968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:19.811097 18968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d982a314a894083a902fc3cbaf3da8d, State: Initialized, Role: FOLLOWER
I20260925 00:53:19.811916 18968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d982a314a894083a902fc3cbaf3da8d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46661 } }
I20260925 00:53:19.812076 18968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:19.812140 18968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:19.812242 18968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:19.813467 18968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d982a314a894083a902fc3cbaf3da8d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46661 } }
I20260925 00:53:19.813827 18968 leader_election.cc:302] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d982a314a894083a902fc3cbaf3da8d; no voters:
I20260925 00:53:19.814178 18968 leader_election.cc:288] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:19.814355 18973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:19.814575 18973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [term 1 LEADER]: Becoming Leader. State: Replica: 5d982a314a894083a902fc3cbaf3da8d, State: Running, Role: LEADER
I20260925 00:53:19.814886 18973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d982a314a894083a902fc3cbaf3da8d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46661 } }
I20260925 00:53:19.815507 18968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:19.816903 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d982a314a894083a902fc3cbaf3da8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d982a314a894083a902fc3cbaf3da8d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46661 } } }
I20260925 00:53:19.817030 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:19.817329 18975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d982a314a894083a902fc3cbaf3da8d. Latest consensus state: current_term: 1 leader_uuid: "5d982a314a894083a902fc3cbaf3da8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d982a314a894083a902fc3cbaf3da8d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46661 } } }
I20260925 00:53:19.817454 18975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d [sys.catalog]: This master's current role is: LEADER
W20260925 00:53:19.820281 18989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:53:19.820356 18989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:53:19.820462 18983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:19.821197 18983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:19.826022 18983 catalog_manager.cc:1347] Generated new cluster ID: 6638e0696f9c42db95f420ab86e4e0f0
I20260925 00:53:19.826094 18983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:19.851328 18983 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:19.852371 18983 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:19.856602 18983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Generated new TSK 0
I20260925 00:53:19.857371 18983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:19.892660 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:46661
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:20.008456 18994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:20.008895 18994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:20.008989 18994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:20.013288 18994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:20.013388 18994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:20.013456 18994 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 00:53:20.013489 18994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:20.013564 18994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:20.019412 18994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:46661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:20.020597 18994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:20.021724 18994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:20.031594 19000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:20.031596 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:20.031596 18999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:20.032066 18994 server_base.cc:1034] running on GCE node
I20260925 00:53:20.032507 18994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:20.033316 18994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:20.034493 18994 hybrid_clock.cc:648] HybridClock initialized: now 1790297600034470 us; error 41 us; skew 500 ppm
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:20.037335 18994 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:20.038275 18994 webserver.cc:460] Webserver started at http://127.16.202.1:46333/ using document root <none> and password file <none>
I20260925 00:53:20.038915 18994 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:20.038982 18994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:20.039155 18994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:20.041400 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "aeef965e9b0443c88642d36c09c6be7a"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "2a51476706bed78898add936e0aa4f01"
server_key_iv: "8279f1229376bcd8fe4318667e5157db"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.041970 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "aeef965e9b0443c88642d36c09c6be7a"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "2a51476706bed78898add936e0aa4f01"
server_key_iv: "8279f1229376bcd8fe4318667e5157db"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.045960 18994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 00:53:20.048929 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.050411 18994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:53:20.050536 18994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "aeef965e9b0443c88642d36c09c6be7a"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "2a51476706bed78898add936e0aa4f01"
server_key_iv: "8279f1229376bcd8fe4318667e5157db"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.050666 18994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:20.075555 18994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:20.078886 18994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:20.079064 18994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:20.080650 18994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:20.080747 18994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.080801 18994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:20.080832 18994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:20.090858 18994 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:34243
I20260925 00:53:20.090904 19124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:34243 every 8 connection(s)
I20260925 00:53:20.091756 18994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:20.094718 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.081574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34737 (local address 127.16.202.62:46661)
0925 00:53:20.081722 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.081727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.082628 (+ 901us) server_negotiation.cc:407] Connection header received
0925 00:53:20.083664 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.083668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.083724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.083809 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.085472 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.086054 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.087715 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.088072 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.088160 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.091167 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.091171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.091174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.091207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.092729 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.093226 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.093229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.093231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.093326 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.093633 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.093636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.093638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.093841 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.093930 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.094417 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.094551 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:53:20.095504 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.081992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.16.202.1:34737)
0925 00:53:20.082456 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.082508 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.083451 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.084035 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.084045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.084527 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.085309 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.085331 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.086161 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.086166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.087581 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.087595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.088225 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.088229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.088441 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.089162 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.089184 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.090966 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.092832 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.092838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.092854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.093120 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.093406 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.093410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.093412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.093544 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.093939 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.093947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.094188 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.094355 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.094676 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
I20260925 00:53:20.096808 19125 heartbeater.cc:347] Connected to a master server at 127.16.202.62:46661
I20260925 00:53:20.096958 19125 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:20.097527 19125 heartbeater.cc:511] Master 127.16.202.62:46661 requested a full tablet report, sending...
I20260925 00:53:20.099402 18921 ts_manager.cc:194] Registered new tserver with Master: aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243)
I20260925 00:53:20.100409 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 18994
I20260925 00:53:20.100534 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:20.100878 17192 external_mini_cluster.cc:1428] Setting key 007b6d4d2c94fda2b287f31cca80652b
I20260925 00:53:20.100891 18921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:34737
I20260925 00:53:20.107167 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.099089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.16.202.1:54889)
0925 00:53:20.099479 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.099494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.099598 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.099821 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.099824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.100048 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.100173 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.100181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.101239 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.101250 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.102094 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.102105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.102827 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.102830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.102914 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.103438 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.103449 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.103807 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.105690 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.105693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.105694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.105912 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.106161 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.106163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.106164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.106235 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.106682 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.106684 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.106784 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.106876 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.107006 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":162,"threads_started":1}
I20260925 00:53:20.107250 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.099194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:54889 (local address 127.16.202.62:46661)
0925 00:53:20.099323 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.099327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.099585 (+ 258us) server_negotiation.cc:407] Connection header received
0925 00:53:20.099674 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.099676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.099719 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.099799 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.100293 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.101106 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.102228 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.102607 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.102679 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.103954 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.103959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.103962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.103992 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.105577 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.105997 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.106000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.106002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.106088 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.106302 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.106305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.106307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.106563 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.106696 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.106948 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.107085 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:20.128551 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:46661
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:20.241696 19134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:20.241972 19134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:20.242020 19134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:20.246150 19134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:20.246224 19134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:20.246268 19134 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 00:53:20.246286 19134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:20.246323 19134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:20.251830 19134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:46661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:20.253027 19134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:20.254072 19134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:20.263927 19140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:20.264034 19134 server_base.cc:1034] running on GCE node
W20260925 00:53:20.263917 19139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:20.263958 19142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:20.264701 19134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:20.265479 19134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:20.266683 19134 hybrid_clock.cc:648] HybridClock initialized: now 1790297600266635 us; error 61 us; skew 500 ppm
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:20.269819 19134 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:20.270844 19134 webserver.cc:460] Webserver started at http://127.16.202.2:46651/ using document root <none> and password file <none>
I20260925 00:53:20.271414 19134 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:20.271476 19134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:20.271660 19134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:20.273856 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "0c598e2ffed94363a02d0d9cf8b56594"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "bc72395b6c06d465bb025d2b221340ac"
server_key_iv: "9b8bb9b517451fd5b3fd17ab16fa4d22"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.274405 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "0c598e2ffed94363a02d0d9cf8b56594"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "bc72395b6c06d465bb025d2b221340ac"
server_key_iv: "9b8bb9b517451fd5b3fd17ab16fa4d22"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.278645 19134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:20.281427 19149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.282833 19134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:53:20.282956 19134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "0c598e2ffed94363a02d0d9cf8b56594"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "bc72395b6c06d465bb025d2b221340ac"
server_key_iv: "9b8bb9b517451fd5b3fd17ab16fa4d22"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.283094 19134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:20.301936 19134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:20.305331 19134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:20.305513 19134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:20.307228 19134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:20.307297 19134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.307355 19134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:20.307377 19134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.317572 19134 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:37599
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:20.318544 19134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:20.319108 19264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:37599 every 8 connection(s)
I20260925 00:53:20.322206 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.308000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:43087 (local address 127.16.202.62:46661)
0925 00:53:20.308152 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.308157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.309537 (+ 1380us) server_negotiation.cc:407] Connection header received
0925 00:53:20.310807 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.310810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.310872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.310956 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.312710 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.313271 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.314967 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.315189 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.315272 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.318435 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.318439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.318442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.318471 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.319963 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.320468 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.320471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.320473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.320558 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.320921 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.320925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.320927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.321125 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.321216 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.321870 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.322033 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:20.323323 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.308875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.16.202.2:43087)
0925 00:53:20.309359 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.309423 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.310559 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.311193 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.311204 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.311737 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.312495 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.312518 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.313392 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.313396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.314831 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.314843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.315334 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.315339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.315568 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.316320 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.316343 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.318246 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.320082 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.320088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.320105 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.320360 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.320643 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.320647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.320649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.320828 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.321225 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.321232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.321495 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.321675 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.322064 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260925 00:53:20.324549 19265 heartbeater.cc:347] Connected to a master server at 127.16.202.62:46661
I20260925 00:53:20.324753 19265 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:20.325372 19265 heartbeater.cc:511] Master 127.16.202.62:46661 requested a full tablet report, sending...
I20260925 00:53:20.325489 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 19134
I20260925 00:53:20.325596 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:20.325865 17192 external_mini_cluster.cc:1428] Setting key 96581371462cfe4f9128770108396a86
I20260925 00:53:20.326592 18922 ts_manager.cc:194] Registered new tserver with Master: 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599)
I20260925 00:53:20.327258 18922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:43087
I20260925 00:53:20.335126 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.327614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:52517 (local address 127.16.202.62:46661)
0925 00:53:20.327722 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.327726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.327738 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:53:20.327773 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.327776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.327816 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.327898 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.328453 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.329173 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.330063 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.330285 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.330374 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.331551 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.331554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.331557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.331580 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.333187 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.333743 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.333746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.333748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.333847 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.334170 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.334173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.334174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.334476 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.334566 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.334886 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.334999 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:53:20.335206 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.327034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.16.202.2:52517)
0925 00:53:20.327376 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.327392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.327477 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.327937 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.327940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.328161 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.328331 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.328339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.329330 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.329334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.329943 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.329953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.330387 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.330390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.330473 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.330962 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.330976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.331396 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.333311 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.333314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.333317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.333607 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.333941 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.333944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.333946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.334044 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.334580 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.334584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.334698 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.334817 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.334994 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:20.356596 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:46661
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:20.469564 19274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:20.469851 19274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:20.469907 19274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:20.474040 19274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:20.474109 19274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:20.474150 19274 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 00:53:20.474164 19274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:20.474205 19274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:20.479578 19274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:46661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:20.480754 19274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:20.481802 19274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:20.492262 19279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:20.492295 19282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:20.492288 19280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:20.492640 19274 server_base.cc:1034] running on GCE node
I20260925 00:53:20.493091 19274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:20.493788 19274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:20.494937 19274 hybrid_clock.cc:648] HybridClock initialized: now 1790297600494909 us; error 41 us; skew 500 ppm
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:20.497877 19274 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:20.498852 19274 webserver.cc:460] Webserver started at http://127.16.202.3:36651/ using document root <none> and password file <none>
I20260925 00:53:20.499462 19274 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:20.499536 19274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:20.499739 19274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:20.501945 19274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "793b90c55d514ded9313fd34884030f3"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "2d089d5e35cc196ba0212647ae861cec"
server_key_iv: "196ffafed7662e9ba4caf0e04b33cc04"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.502556 19274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "793b90c55d514ded9313fd34884030f3"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "2d089d5e35cc196ba0212647ae861cec"
server_key_iv: "196ffafed7662e9ba4caf0e04b33cc04"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.506637 19274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:20.509337 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.510807 19274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:20.510936 19274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "793b90c55d514ded9313fd34884030f3"
format_stamp: "Formatted at 2026-09-25 00:53:20 on dist-test-slave-rpk1"
server_key: "2d089d5e35cc196ba0212647ae861cec"
server_key_iv: "196ffafed7662e9ba4caf0e04b33cc04"
server_key_version: "encryptionkey@0"
I20260925 00:53:20.511055 19274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:20.521169 19274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:20.524344 19274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:20.524524 19274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:20.526074 19274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:20.526134 19274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:20.526187 19274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:20.526213 19274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:20.536063 19274 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:46441
I20260925 00:53:20.536092 19404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:46441 every 8 connection(s)
I20260925 00:53:20.536969 19274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:20.539211 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.527018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:47499 (local address 127.16.202.62:46661)
0925 00:53:20.527135 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.527139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.528190 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 00:53:20.529426 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.529430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.529488 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.529562 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.530824 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.531408 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.532528 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.532840 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.532950 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.535640 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.535644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.535647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.535675 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.537182 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.537743 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.537747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.537749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.537830 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.538145 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.538148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.538150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.538344 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.538434 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.538921 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.539043 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:53:20.539986 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.527533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.16.202.3:47499)
0925 00:53:20.528032 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.528075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.529188 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.529746 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.529753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.530087 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.530677 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.530692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.531517 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.531520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.532384 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.532396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.533011 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.533015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.533186 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.533740 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.533757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.535471 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.537293 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.537298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.537314 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.537620 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.537911 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.537914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.537916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.538043 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.538444 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.538451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.538682 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.538878 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.539190 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260925 00:53:20.541093 19405 heartbeater.cc:347] Connected to a master server at 127.16.202.62:46661
I20260925 00:53:20.541236 19405 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:20.541723 19405 heartbeater.cc:511] Master 127.16.202.62:46661 requested a full tablet report, sending...
I20260925 00:53:20.542788 18921 ts_manager.cc:194] Registered new tserver with Master: 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441)
I20260925 00:53:20.543426 18921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:47499
I20260925 00:53:20.543460 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 19274
I20260925 00:53:20.543576 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:20.543833 17192 external_mini_cluster.cc:1428] Setting key 0722b7741fe633418a0b0c6d84ac36c6
I20260925 00:53:20.545900 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:53:20.551030 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.543440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.16.202.3:43423)
0925 00:53:20.544239 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.544255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.544360 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.544573 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.544576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.545013 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.545133 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.545141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.545930 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.545932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.546506 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.546514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.547027 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.547029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.547093 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.547574 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.547584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.547905 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.549649 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.549652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.549654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.549854 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.550129 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.550131 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.550132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.550195 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.550589 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.550592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.550672 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.550750 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.550861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":96,"threads_started":1}
I20260925 00:53:20.551102 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.543357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:43423 (local address 127.16.202.62:46661)
0925 00:53:20.543494 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.543498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.544322 (+ 824us) server_negotiation.cc:407] Connection header received
0925 00:53:20.544436 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.544440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.544483 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.544548 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.545234 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.545798 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.546619 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.546920 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.547017 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.548038 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.548043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.548046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.548072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.549548 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.549956 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.549960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.549961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.550048 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.550261 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.550264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.550266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.550503 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.550602 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.550819 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.550944 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:20.571561 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.563800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.16.202.62:46661)
0925 00:53:20.563929 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.563932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.563946 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:20.563991 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.563993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.564032 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.564109 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.564830 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.565383 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.566190 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.566429 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.566512 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.568754 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.568757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.568759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.568814 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.570159 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.570568 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.570571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.570573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.570664 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.570869 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.570871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.570872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.571049 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.571147 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.571321 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.571435 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:53:20.574528 18921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:20.577507 18921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:20.592093 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.586554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:34243 (local address 127.0.0.1:40700)
0925 00:53:20.587235 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.587247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.587352 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.587999 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.588002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.588018 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.588097 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.588102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.589393 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.589396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.590408 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.590418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.591680 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.591684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.591769 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.591795 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.591868 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.591946 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260925 00:53:20.592155 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.585803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37599 (local address 127.0.0.1:33734)
0925 00:53:20.586160 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.586209 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.586351 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.587657 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.587660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.587677 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.587793 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.587800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.588812 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.588815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.590236 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.590248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.591397 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.591401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.591537 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.591582 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.591673 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.591804 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":25,"thread_start_us":80,"threads_started":1}
I20260925 00:53:20.592576 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.586751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.16.202.1:34243)
0925 00:53:20.587111 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.587124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.587437 (+ 313us) server_negotiation.cc:407] Connection header received
0925 00:53:20.587562 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.587566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.587743 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.587868 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.588222 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.589293 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.590510 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.591318 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.591425 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.591614 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.592313 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.592431 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":70,"threads_started":1}
I20260925 00:53:20.592844 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.586775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.16.202.2:37599)
0925 00:53:20.587179 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.587190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.587212 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:53:20.587281 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.587285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.587528 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.587678 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.587905 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.588663 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.590387 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.591247 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.592337 (+ 1090us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.592523 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.592573 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.592707 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":113,"threads_started":1}
I20260925 00:53:20.598703 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.585846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:46441 (local address 127.0.0.1:57256)
0925 00:53:20.586217 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.586230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.586347 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.587421 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.587428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.587449 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.587604 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.587611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.589114 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.589118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.596849 (+ 7731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.596864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.598281 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.598285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.598396 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.598426 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.598472 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.598551 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":53,"thread_start_us":116,"threads_started":1}
I20260925 00:53:20.599586 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.586504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.16.202.3:46441)
0925 00:53:20.586917 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.586930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.586954 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:53:20.587031 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.587036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.587255 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.587745 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.587892 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.589003 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.597021 (+ 8018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.598134 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.598966 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.599290 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.599348 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.599455 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":96,"threads_started":1}
I20260925 00:53:20.602270 19059 tablet_service.cc:1467] Processing CreateTablet for tablet f1980e26f757471098f783ae1970651f (DEFAULT_TABLE table=test-table [id=ca52656cda9549dca4fa316dc9438d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:20.602279 19339 tablet_service.cc:1467] Processing CreateTablet for tablet f1980e26f757471098f783ae1970651f (DEFAULT_TABLE table=test-table [id=ca52656cda9549dca4fa316dc9438d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:20.603577 19339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1980e26f757471098f783ae1970651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:20.603574 19059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1980e26f757471098f783ae1970651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:20.604746 19199 tablet_service.cc:1467] Processing CreateTablet for tablet f1980e26f757471098f783ae1970651f (DEFAULT_TABLE table=test-table [id=ca52656cda9549dca4fa316dc9438d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:20.605894 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1980e26f757471098f783ae1970651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:20.612190 19428 tablet_bootstrap.cc:492] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Bootstrap starting.
I20260925 00:53:20.615028 19428 tablet_bootstrap.cc:654] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:20.615145 19429 tablet_bootstrap.cc:492] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Bootstrap starting.
I20260925 00:53:20.616132 19430 tablet_bootstrap.cc:492] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Bootstrap starting.
I20260925 00:53:20.616120 19428 log.cc:824] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:20.618126 19429 tablet_bootstrap.cc:654] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:20.618513 19430 tablet_bootstrap.cc:654] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:20.618767 19428 tablet_bootstrap.cc:492] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: No bootstrap required, opened a new log
I20260925 00:53:20.618979 19428 ts_tablet_manager.cc:1397] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:53:20.619343 19429 log.cc:824] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:20.619356 19430 log.cc:824] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:20.621388 19429 tablet_bootstrap.cc:492] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: No bootstrap required, opened a new log
I20260925 00:53:20.621409 19430 tablet_bootstrap.cc:492] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: No bootstrap required, opened a new log
I20260925 00:53:20.621580 19429 ts_tablet_manager.cc:1397] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260925 00:53:20.621623 19430 ts_tablet_manager.cc:1397] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 00:53:20.623237 19428 raft_consensus.cc:348] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.623544 19428 raft_consensus.cc:374] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:20.623608 19428 raft_consensus.cc:729] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Initialized, Role: FOLLOWER
I20260925 00:53:20.624087 19428 consensus_queue.cc:260] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.624853 19405 heartbeater.cc:503] Master 127.16.202.62:46661 was elected leader, sending a full tablet report...
I20260925 00:53:20.625301 19428 ts_tablet_manager.cc:1428] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:53:20.624660 19429 raft_consensus.cc:348] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.626009 19429 raft_consensus.cc:374] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:20.626065 19429 raft_consensus.cc:729] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeef965e9b0443c88642d36c09c6be7a, State: Initialized, Role: FOLLOWER
I20260925 00:53:20.626480 19429 consensus_queue.cc:260] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.627106 19125 heartbeater.cc:503] Master 127.16.202.62:46661 was elected leader, sending a full tablet report...
I20260925 00:53:20.627770 19429 ts_tablet_manager.cc:1428] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260925 00:53:20.629212 19430 raft_consensus.cc:348] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.629501 19430 raft_consensus.cc:374] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:20.629557 19430 raft_consensus.cc:729] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Initialized, Role: FOLLOWER
I20260925 00:53:20.630051 19430 consensus_queue.cc:260] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.630735 19265 heartbeater.cc:503] Master 127.16.202.62:46661 was elected leader, sending a full tablet report...
I20260925 00:53:20.631145 19430 ts_tablet_manager.cc:1428] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20260925 00:53:20.634969 19434 raft_consensus.cc:482] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:20.635140 19434 raft_consensus.cc:504] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.636315 19434 leader_election.cc:288] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599), aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243)
I20260925 00:53:20.641944 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.637285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:50847 (local address 127.16.202.1:34243)
0925 00:53:20.637419 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.637425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.637442 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:53:20.637494 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.637497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.637546 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.637651 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.637979 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.638781 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.640797 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.641314 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.641425 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.641469 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.641728 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.641811 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:53:20.642144 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.636758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:34243 (local address 127.16.202.3:50847)
0925 00:53:20.637168 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.637182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.637274 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.637668 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.637689 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.637710 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.637854 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.637862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.638920 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.638924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.640040 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.640053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.641440 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.641444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.641607 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.641629 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.641694 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.641749 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":37,"thread_start_us":134,"threads_started":1}
I20260925 00:53:20.642771 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1980e26f757471098f783ae1970651f" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a" is_pre_election: true
I20260925 00:53:20.643054 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.636491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37599 (local address 127.16.202.3:59479)
0925 00:53:20.636611 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.636627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.636749 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.639735 (+ 2986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.639738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.639759 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.639854 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.639860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.640767 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.640823 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.641876 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.641887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.642718 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.642720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.642792 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.642810 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.642866 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.642932 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":27}
I20260925 00:53:20.643075 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.637173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:59479 (local address 127.16.202.2:37599)
0925 00:53:20.639436 (+ 2263us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.639441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.639456 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:20.639503 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.639506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.639551 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.639629 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.639979 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.640578 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.642031 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.642603 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.642736 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.642791 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.642884 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.642957 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2170}
I20260925 00:53:20.643132 19067 raft_consensus.cc:2454] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 0.
I20260925 00:53:20.643613 19291 leader_election.cc:302] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793b90c55d514ded9313fd34884030f3, aeef965e9b0443c88642d36c09c6be7a; no voters:
I20260925 00:53:20.643652 19219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1980e26f757471098f783ae1970651f" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c598e2ffed94363a02d0d9cf8b56594" is_pre_election: true
I20260925 00:53:20.643873 19434 raft_consensus.cc:2781] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:20.643906 19219 raft_consensus.cc:2454] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 0.
I20260925 00:53:20.643966 19434 raft_consensus.cc:482] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:20.644021 19434 raft_consensus.cc:3037] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:20.645298 19434 raft_consensus.cc:504] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.645633 19434 leader_election.cc:288] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 election: Requested vote from peers 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599), aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243)
I20260925 00:53:20.645951 19219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1980e26f757471098f783ae1970651f" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c598e2ffed94363a02d0d9cf8b56594"
I20260925 00:53:20.646060 19219 raft_consensus.cc:3037] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:20.646075 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1980e26f757471098f783ae1970651f" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a"
I20260925 00:53:20.646211 19067 raft_consensus.cc:3037] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:20.647058 19219 raft_consensus.cc:2454] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 1.
I20260925 00:53:20.647226 19067 raft_consensus.cc:2454] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 1.
I20260925 00:53:20.647336 19291 leader_election.cc:302] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, 793b90c55d514ded9313fd34884030f3; no voters:
I20260925 00:53:20.647507 19434 raft_consensus.cc:2781] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:20.647733 19434 raft_consensus.cc:686] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 1 LEADER]: Becoming Leader. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Running, Role: LEADER
I20260925 00:53:20.648043 19434 consensus_queue.cc:237] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } }
I20260925 00:53:20.651053 18921 catalog_manager.cc:5495] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793b90c55d514ded9313fd34884030f3 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "793b90c55d514ded9313fd34884030f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } health_report { overall_health: UNKNOWN } } }
Sep 25 00:53:20 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297599, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:20.691233 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.682603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.16.202.62:46661)
0925 00:53:20.682749 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.682754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.682901 (+ 147us) server_negotiation.cc:407] Connection header received
0925 00:53:20.683039 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.683042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.683079 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.683140 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:20.683799 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.684306 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.685134 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.685338 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.685431 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.688212 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:20.688216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:20.688218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:20.688241 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:20.689580 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.690045 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.690048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.690050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.690121 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:20.690351 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:20.690355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:20.690356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:20.690548 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:20.690632 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.690928 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.691060 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":54}
I20260925 00:53:20.691265 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.682499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46661 (local address 127.0.0.1:59008)
0925 00:53:20.682816 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.682828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.682957 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.683174 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.683177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.683592 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:20.683685 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.683691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.684427 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.684431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.685027 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.685038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.685443 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.685446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.685588 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.686247 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:20.686264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:20.688047 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:20.689691 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.689704 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.689711 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.689956 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.690199 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:20.690202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:20.690204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:20.690279 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:20.690657 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:20.690666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:20.690776 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.690882 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.691138 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":143,"thread_start_us":128,"threads_started":1}
I20260925 00:53:20.695420 18921 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:59008:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 00:53:20.696218 18921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:20.701731 19059 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9fb7fc26c947d6b38cf6a0a88dbf96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03f12dfd16164800b950ecb491bc9a03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:20.701922 19199 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9fb7fc26c947d6b38cf6a0a88dbf96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03f12dfd16164800b950ecb491bc9a03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:20.701732 19339 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9fb7fc26c947d6b38cf6a0a88dbf96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03f12dfd16164800b950ecb491bc9a03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:20.702561 19059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9fb7fc26c947d6b38cf6a0a88dbf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:20.702560 19339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9fb7fc26c947d6b38cf6a0a88dbf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:20.702560 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9fb7fc26c947d6b38cf6a0a88dbf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:20.705401 19429 tablet_bootstrap.cc:492] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: Bootstrap starting.
I20260925 00:53:20.705657 19428 tablet_bootstrap.cc:492] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: Bootstrap starting.
I20260925 00:53:20.705834 19430 tablet_bootstrap.cc:492] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: Bootstrap starting.
I20260925 00:53:20.706462 19429 tablet_bootstrap.cc:654] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:20.706547 19428 tablet_bootstrap.cc:654] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:20.706640 19430 tablet_bootstrap.cc:654] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:20.707778 19430 tablet_bootstrap.cc:492] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: No bootstrap required, opened a new log
I20260925 00:53:20.707778 19429 tablet_bootstrap.cc:492] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: No bootstrap required, opened a new log
I20260925 00:53:20.707799 19428 tablet_bootstrap.cc:492] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: No bootstrap required, opened a new log
I20260925 00:53:20.707860 19430 ts_tablet_manager.cc:1397] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:20.707860 19429 ts_tablet_manager.cc:1397] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 00:53:20.707886 19428 ts_tablet_manager.cc:1397] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:20.708248 19428 raft_consensus.cc:348] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.708334 19428 raft_consensus.cc:374] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:20.708292 19429 raft_consensus.cc:348] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.708292 19430 raft_consensus.cc:348] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.708388 19428 raft_consensus.cc:729] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Initialized, Role: FOLLOWER
I20260925 00:53:20.708406 19430 raft_consensus.cc:374] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:20.708406 19429 raft_consensus.cc:374] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:20.708441 19429 raft_consensus.cc:729] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeef965e9b0443c88642d36c09c6be7a, State: Initialized, Role: FOLLOWER
I20260925 00:53:20.708441 19430 raft_consensus.cc:729] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Initialized, Role: FOLLOWER
I20260925 00:53:20.708495 19428 consensus_queue.cc:260] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.708562 19430 consensus_queue.cc:260] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.708562 19429 consensus_queue.cc:260] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.708792 19428 ts_tablet_manager.cc:1428] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:20.708846 19429 ts_tablet_manager.cc:1428] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:53:20.708843 19430 ts_tablet_manager.cc:1428] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:53:20.708859 19436 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:20.709031 19436 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:20.708987 19434 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:20.709082 19434 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:20.709226 19435 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:20.709347 19435 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 00:53:20.787891 19406 tablet.cc:2369] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:20.820194 19266 tablet.cc:2369] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:20.847607 19126 tablet.cc:2369] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:20.931469 19436 raft_consensus.cc:482] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:20.931636 19436 raft_consensus.cc:504] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.932880 19436 leader_election.cc:288] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441)
I20260925 00:53:20.936757 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.933124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:34243 (local address 127.16.202.2:37245)
0925 00:53:20.933494 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.933506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.933620 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.933870 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.933873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.933887 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.934012 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.934018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.934742 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.934746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.935603 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.935612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.936243 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.936246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.936338 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.936361 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.936475 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.936559 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":8,"spinlock_wait_cycles":12416,"thread_start_us":101,"threads_started":1}
I20260925 00:53:20.936762 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.933266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:37245 (local address 127.16.202.1:34243)
0925 00:53:20.933381 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.933386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.933593 (+ 207us) server_negotiation.cc:407] Connection header received
0925 00:53:20.933729 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.933733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.933778 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.933833 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.934125 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.934622 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.935731 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.936122 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.936238 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.936295 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.936512 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.936603 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:53:20.936805 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.933234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:60801 (local address 127.16.202.3:46441)
0925 00:53:20.933381 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:20.933387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:20.933650 (+ 263us) server_negotiation.cc:407] Connection header received
0925 00:53:20.933761 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:20.933764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:20.933804 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:20.933878 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:20.934133 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.934652 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.935725 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.936119 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.936238 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.936295 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:20.936605 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 00:53:20.936705 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 00:53:20.937134 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:20.933127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:46441 (local address 127.16.202.2:60801)
0925 00:53:20.933540 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:20.933552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:20.933656 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:20.933896 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:20.933899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:20.933933 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:20.934028 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.934032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.934761 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.934764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.935604 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:20.935613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:20.936375 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:20.936379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:20.936469 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:20.936492 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:20.936912 (+ 420us) client_negotiation.cc:240] Negotiation successful
0925 00:53:20.936998 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":107,"spinlock_wait_cycles":1792,"thread_start_us":77,"threads_started":1}
I20260925 00:53:20.937438 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9fb7fc26c947d6b38cf6a0a88dbf96" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a" is_pre_election: true
I20260925 00:53:20.937625 19067 raft_consensus.cc:2454] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 0.
I20260925 00:53:20.937733 19359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9fb7fc26c947d6b38cf6a0a88dbf96" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793b90c55d514ded9313fd34884030f3" is_pre_election: true
I20260925 00:53:20.937983 19359 raft_consensus.cc:2454] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 0.
I20260925 00:53:20.938061 19151 leader_election.cc:302] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, aeef965e9b0443c88642d36c09c6be7a; no voters:
I20260925 00:53:20.938369 19436 raft_consensus.cc:2781] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:20.938441 19436 raft_consensus.cc:482] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:20.938476 19436 raft_consensus.cc:3037] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:20.939499 19436 raft_consensus.cc:504] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.939850 19436 leader_election.cc:288] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 election: Requested vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441)
I20260925 00:53:20.940213 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9fb7fc26c947d6b38cf6a0a88dbf96" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a"
I20260925 00:53:20.940271 19359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9fb7fc26c947d6b38cf6a0a88dbf96" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793b90c55d514ded9313fd34884030f3"
I20260925 00:53:20.940346 19067 raft_consensus.cc:3037] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:20.940375 19359 raft_consensus.cc:3037] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:20.941395 19067 raft_consensus.cc:2454] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 1.
I20260925 00:53:20.941401 19359 raft_consensus.cc:2454] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 1.
I20260925 00:53:20.941682 19150 leader_election.cc:302] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, 793b90c55d514ded9313fd34884030f3; no voters:
I20260925 00:53:20.941860 19436 raft_consensus.cc:2781] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:20.942037 19436 raft_consensus.cc:686] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 LEADER]: Becoming Leader. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Running, Role: LEADER
I20260925 00:53:20.942387 19436 consensus_queue.cc:237] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:20.944424 19451 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: TxnStatusTablet state changed. Reason: New leader 0c598e2ffed94363a02d0d9cf8b56594. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:20.944547 19451 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:20.944862 19453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 00:53:20.945098 19453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 00:53:20.945873 18922 catalog_manager.cc:5495] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2). New cstate: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:21.004154 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.000531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37599 (local address 127.0.0.1:33736)
0925 00:53:21.000707 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:21.000723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:21.000833 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:21.001158 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:21.001161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:21.001168 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:21.001275 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:21.001280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.002001 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.002005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:21.002609 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:21.002618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.003044 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.003048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:21.003139 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.003187 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.003862 (+ 675us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.003879 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:53:21.003943 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:53:21.004033 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 00:53:21.004154 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.000661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.16.202.2:37599)
0925 00:53:21.000824 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:21.000829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:21.000843 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:21.000988 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:21.000991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:21.001044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:21.001129 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:21.001372 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.001893 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.002723 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.002941 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.003027 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.003326 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.003770 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.003879 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:21.003953 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 00:53:21.004009 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 00:53:21.009262 19359 raft_consensus.cc:1261] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Refusing update from remote peer 0c598e2ffed94363a02d0d9cf8b56594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:21.009362 19067 raft_consensus.cc:1261] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Refusing update from remote peer 0c598e2ffed94363a02d0d9cf8b56594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:21.009884 19451 consensus_queue.cc:1035] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:21.010146 19436 consensus_queue.cc:1035] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:21.013607 19434 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: TxnStatusTablet state changed. Reason: New leader 0c598e2ffed94363a02d0d9cf8b56594. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:21.013731 19434 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:21.014310 19435 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: TxnStatusTablet state changed. Reason: New leader 0c598e2ffed94363a02d0d9cf8b56594. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:21.014420 19435 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:21.017359 19451 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:21.017474 19451 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:21.017592 19451 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:21.017648 19451 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 0c598e2ffed94363a02d0d9cf8b56594: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:21.018204 19456 mvcc.cc:205] Tried to move back new op lower bound from 7333058973726945280 to 7333058973462585344. Current Snapshot: MvccSnapshot[applied={T|T < 7333058973726945280}]
I20260925 00:53:21.018626 19434 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:21.018731 19434 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P 793b90c55d514ded9313fd34884030f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:21.018890 19457 mvcc.cc:205] Tried to move back new op lower bound from 7333058973726945280 to 7333058973462585344. Current Snapshot: MvccSnapshot[applied={T|T < 7333058973726945280}]
I20260925 00:53:21.020021 19435 tablet_replica.cc:439] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } } }
I20260925 00:53:21.020135 19435 tablet_replica.cc:442] T 2d9fb7fc26c947d6b38cf6a0a88dbf96 P aeef965e9b0443c88642d36c09c6be7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:21.021986 19458 mvcc.cc:205] Tried to move back new op lower bound from 7333058973726945280 to 7333058973462585344. Current Snapshot: MvccSnapshot[applied={T|T < 7333058973726945280}]
I20260925 00:53:21.031183 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.027262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.16.202.3:46441)
0925 00:53:21.027422 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:21.027427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:21.027442 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:21.027544 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:21.027547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:21.027609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:21.027714 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:21.027917 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.028744 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.029617 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.029965 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.030064 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.030270 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.030807 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.030925 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:21.030965 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 00:53:21.031020 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 00:53:21.043560 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.039420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37599 (local address 127.16.202.3:50743)
0925 00:53:21.039569 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:21.039587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:21.039698 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:21.039968 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:21.039971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:21.039979 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:21.040108 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:21.040115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.041356 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.041359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:21.042094 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:21.042104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.042705 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.042708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:21.042789 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.042842 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.043275 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.043283 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 00:53:21.043353 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:53:21.043435 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 00:53:21.043598 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.039472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:50743 (local address 127.16.202.2:37599)
0925 00:53:21.039592 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:21.039597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:21.039636 (+ 39us) server_negotiation.cc:407] Connection header received
0925 00:53:21.039791 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:21.039794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:21.039877 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:21.039943 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:21.040238 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.041186 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.042228 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.042577 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.042687 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.043058 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.043160 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.043245 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:21.043370 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 00:53:21.043444 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:53:21.051716 19067 raft_consensus.cc:1261] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Refusing update from remote peer 793b90c55d514ded9313fd34884030f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:21.051821 19219 raft_consensus.cc:1261] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Refusing update from remote peer 793b90c55d514ded9313fd34884030f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:21.052232 19434 consensus_queue.cc:1035] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:21.052448 19438 consensus_queue.cc:1035] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:21.076910 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.073187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:59503 (local address 127.16.202.3:46441)
0925 00:53:21.073292 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:21.073296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:21.073307 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:53:21.073366 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:21.073368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:21.073412 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:21.073469 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:21.073719 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.074199 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.075855 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.076098 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.076171 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.076511 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.076576 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.076623 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:21.076752 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 00:53:21.076802 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 00:53:21.076996 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:21.072844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:46441 (local address 127.16.202.2:59503)
0925 00:53:21.073169 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:21.073185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:21.073286 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:21.073498 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:21.073501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:21.073509 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:21.073630 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:21.073637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.074306 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.074310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:21.075740 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:21.075751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:21.076215 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:21.076218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:21.076334 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:21.076390 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:21.076646 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:21.076653 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:53:21.076745 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:53:21.076829 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260925 00:53:21.104568 18920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:21.104799 18920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:21.107494 18920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Sending DeleteTablet for 3 replicas of tablet f1980e26f757471098f783ae1970651f
I20260925 00:53:21.108541 19339 tablet_service.cc:1514] Processing DeleteTablet for tablet f1980e26f757471098f783ae1970651f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:21 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:57256
I20260925 00:53:21.108518 19059 tablet_service.cc:1514] Processing DeleteTablet for tablet f1980e26f757471098f783ae1970651f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:21 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:40700
I20260925 00:53:21.109207 19199 tablet_service.cc:1514] Processing DeleteTablet for tablet f1980e26f757471098f783ae1970651f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:21 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33734
I20260925 00:53:21.109459 19475 tablet_replica.cc:330] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: stopping tablet replica
I20260925 00:53:21.109804 19475 raft_consensus.cc:2229] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:21.110051 19475 raft_consensus.cc:2258] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:21.110455 18920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:21.110915 18920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:21.111572 19475 ts_tablet_manager.cc:1905] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:21.114871 19475 ts_tablet_manager.cc:1918] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:21.114975 19475 log.cc:1196] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/f1980e26f757471098f783ae1970651f
I20260925 00:53:21.115278 19475 ts_tablet_manager.cc:1939] T f1980e26f757471098f783ae1970651f P aeef965e9b0443c88642d36c09c6be7a: Deleting consensus metadata
I20260925 00:53:21.116321 19476 tablet_replica.cc:330] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: stopping tablet replica
I20260925 00:53:21.116401 19059 tablet_service.cc:1467] Processing CreateTablet for tablet 686171cea65c4394a50a83ed6387c0ad (DEFAULT_TABLE table=test-table [id=6bcf9524fffa47be9d96a424c89c43c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:21.116647 19059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686171cea65c4394a50a83ed6387c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:21.116653 19476 raft_consensus.cc:2229] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:21.117057 19476 raft_consensus.cc:2258] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:21.117725 18907 catalog_manager.cc:4841] TS aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243): tablet f1980e26f757471098f783ae1970651f (table test-table [id=ca52656cda9549dca4fa316dc9438d8e]) successfully deleted
I20260925 00:53:21.118889 19429 tablet_bootstrap.cc:492] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Bootstrap starting.
I20260925 00:53:21.118866 19199 tablet_service.cc:1467] Processing CreateTablet for tablet 686171cea65c4394a50a83ed6387c0ad (DEFAULT_TABLE table=test-table [id=6bcf9524fffa47be9d96a424c89c43c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:21.119091 19476 ts_tablet_manager.cc:1905] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:21.119151 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686171cea65c4394a50a83ed6387c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:21.119773 19429 tablet_bootstrap.cc:654] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:21.120568 19339 tablet_service.cc:1467] Processing CreateTablet for tablet 686171cea65c4394a50a83ed6387c0ad (DEFAULT_TABLE table=test-table [id=6bcf9524fffa47be9d96a424c89c43c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:21.120832 19339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686171cea65c4394a50a83ed6387c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:21.121882 19430 tablet_bootstrap.cc:492] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Bootstrap starting.
I20260925 00:53:21.121986 19476 ts_tablet_manager.cc:1918] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:21.122061 19476 log.cc:1196] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/f1980e26f757471098f783ae1970651f
I20260925 00:53:21.122330 19476 ts_tablet_manager.cc:1939] T f1980e26f757471098f783ae1970651f P 793b90c55d514ded9313fd34884030f3: Deleting consensus metadata
I20260925 00:53:21.122774 19430 tablet_bootstrap.cc:654] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:21.122941 19428 tablet_bootstrap.cc:492] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Bootstrap starting.
I20260925 00:53:21.123761 19428 tablet_bootstrap.cc:654] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:21.123888 19477 tablet_replica.cc:330] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: stopping tablet replica
I20260925 00:53:21.124039 19429 tablet_bootstrap.cc:492] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: No bootstrap required, opened a new log
I20260925 00:53:21.124150 19429 ts_tablet_manager.cc:1397] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 00:53:21.124368 19477 raft_consensus.cc:2229] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:21.124598 19477 raft_consensus.cc:2258] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:21.124595 19429 raft_consensus.cc:348] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.124761 19429 raft_consensus.cc:374] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:21.124809 19429 raft_consensus.cc:729] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeef965e9b0443c88642d36c09c6be7a, State: Initialized, Role: FOLLOWER
I20260925 00:53:21.124966 19429 consensus_queue.cc:260] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.125041 18906 catalog_manager.cc:4841] TS 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441): tablet f1980e26f757471098f783ae1970651f (table test-table [id=ca52656cda9549dca4fa316dc9438d8e]) successfully deleted
I20260925 00:53:21.125238 19429 ts_tablet_manager.cc:1428] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:53:21.126331 19477 ts_tablet_manager.cc:1905] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:21.126514 19430 tablet_bootstrap.cc:492] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: No bootstrap required, opened a new log
I20260925 00:53:21.126611 19430 ts_tablet_manager.cc:1397] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 00:53:21.127050 19430 raft_consensus.cc:348] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.127170 19430 raft_consensus.cc:374] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:21.127208 19430 raft_consensus.cc:729] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Initialized, Role: FOLLOWER
I20260925 00:53:21.127323 19430 consensus_queue.cc:260] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.127592 19430 ts_tablet_manager.cc:1428] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:53:21.128892 19428 tablet_bootstrap.cc:492] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: No bootstrap required, opened a new log
I20260925 00:53:21.128991 19428 ts_tablet_manager.cc:1397] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 00:53:21.129460 19428 raft_consensus.cc:348] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.129575 19428 raft_consensus.cc:374] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:21.129613 19428 raft_consensus.cc:729] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Initialized, Role: FOLLOWER
I20260925 00:53:21.129745 19428 consensus_queue.cc:260] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.130010 19428 ts_tablet_manager.cc:1428] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:21.130143 19477 ts_tablet_manager.cc:1918] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:21.130255 19477 log.cc:1196] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/f1980e26f757471098f783ae1970651f
I20260925 00:53:21.130606 19477 ts_tablet_manager.cc:1939] T f1980e26f757471098f783ae1970651f P 0c598e2ffed94363a02d0d9cf8b56594: Deleting consensus metadata
I20260925 00:53:21.131932 18907 catalog_manager.cc:4841] TS 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599): tablet f1980e26f757471098f783ae1970651f (table test-table [id=ca52656cda9549dca4fa316dc9438d8e]) successfully deleted
I20260925 00:53:21.237427 19470 raft_consensus.cc:482] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:21.237597 19470 raft_consensus.cc:504] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.238123 19470 leader_election.cc:288] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599)
I20260925 00:53:21.238595 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686171cea65c4394a50a83ed6387c0ad" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a" is_pre_election: true
I20260925 00:53:21.238754 19067 raft_consensus.cc:2454] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 0.
I20260925 00:53:21.238732 19219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686171cea65c4394a50a83ed6387c0ad" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c598e2ffed94363a02d0d9cf8b56594" is_pre_election: true
I20260925 00:53:21.238922 19219 raft_consensus.cc:2454] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 0.
I20260925 00:53:21.239053 19291 leader_election.cc:302] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793b90c55d514ded9313fd34884030f3, aeef965e9b0443c88642d36c09c6be7a; no voters:
I20260925 00:53:21.239225 19470 raft_consensus.cc:2781] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:21.239282 19470 raft_consensus.cc:482] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:21.239315 19470 raft_consensus.cc:3037] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:21.240819 19470 raft_consensus.cc:504] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.241259 19470 leader_election.cc:288] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 election: Requested vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599)
I20260925 00:53:21.241708 19219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686171cea65c4394a50a83ed6387c0ad" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c598e2ffed94363a02d0d9cf8b56594"
I20260925 00:53:21.241753 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686171cea65c4394a50a83ed6387c0ad" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a"
I20260925 00:53:21.241834 19219 raft_consensus.cc:3037] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:21.241891 19067 raft_consensus.cc:3037] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:21.242929 19219 raft_consensus.cc:2454] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 1.
I20260925 00:53:21.243050 19067 raft_consensus.cc:2454] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 1.
I20260925 00:53:21.243273 19291 leader_election.cc:302] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, 793b90c55d514ded9313fd34884030f3; no voters:
I20260925 00:53:21.243434 19470 raft_consensus.cc:2781] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:21.243517 19470 raft_consensus.cc:686] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 1 LEADER]: Becoming Leader. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Running, Role: LEADER
I20260925 00:53:21.243654 19470 consensus_queue.cc:237] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:21.245430 18922 catalog_manager.cc:5495] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793b90c55d514ded9313fd34884030f3 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "793b90c55d514ded9313fd34884030f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } health_report { overall_health: HEALTHY } } }
I20260925 00:53:21.269280 19067 raft_consensus.cc:1261] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Refusing update from remote peer 793b90c55d514ded9313fd34884030f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:21.269268 19219 raft_consensus.cc:1261] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Refusing update from remote peer 793b90c55d514ded9313fd34884030f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:21.269639 19470 consensus_queue.cc:1035] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:21.269819 19434 consensus_queue.cc:1035] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:53:21.283196 19457 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 00:53:22.834620 18922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:22.834780 18922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:22.837396 18922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Sending DeleteTablet for 3 replicas of tablet 686171cea65c4394a50a83ed6387c0ad
I20260925 00:53:22.838016 19059 tablet_service.cc:1514] Processing DeleteTablet for tablet 686171cea65c4394a50a83ed6387c0ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:22 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:40700
I20260925 00:53:22.838138 19199 tablet_service.cc:1514] Processing DeleteTablet for tablet 686171cea65c4394a50a83ed6387c0ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:22 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33734
I20260925 00:53:22.838351 19339 tablet_service.cc:1514] Processing DeleteTablet for tablet 686171cea65c4394a50a83ed6387c0ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:22 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:57256
I20260925 00:53:22.838408 19506 tablet_replica.cc:330] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: stopping tablet replica
I20260925 00:53:22.838541 19506 raft_consensus.cc:2229] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:22.838584 19507 tablet_replica.cc:330] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: stopping tablet replica
I20260925 00:53:22.838672 19506 raft_consensus.cc:2258] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:22.838763 19507 raft_consensus.cc:2229] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:22.838843 19508 tablet_replica.cc:330] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: stopping tablet replica
I20260925 00:53:22.838899 19507 raft_consensus.cc:2258] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:22.838956 19508 raft_consensus.cc:2229] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:22.839162 19508 raft_consensus.cc:2258] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:22.839522 19507 ts_tablet_manager.cc:1905] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:22.839744 19508 ts_tablet_manager.cc:1905] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:22.842065 19507 ts_tablet_manager.cc:1918] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:53:22.842140 19507 log.cc:1196] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/686171cea65c4394a50a83ed6387c0ad
I20260925 00:53:22.842434 19507 ts_tablet_manager.cc:1939] T 686171cea65c4394a50a83ed6387c0ad P 0c598e2ffed94363a02d0d9cf8b56594: Deleting consensus metadata
I20260925 00:53:22.842417 19508 ts_tablet_manager.cc:1918] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:53:22.842489 19508 log.cc:1196] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/686171cea65c4394a50a83ed6387c0ad
I20260925 00:53:22.842753 19508 ts_tablet_manager.cc:1939] T 686171cea65c4394a50a83ed6387c0ad P 793b90c55d514ded9313fd34884030f3: Deleting consensus metadata
I20260925 00:53:22.843155 18907 catalog_manager.cc:4841] TS 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599): tablet 686171cea65c4394a50a83ed6387c0ad (table test-table [id=6bcf9524fffa47be9d96a424c89c43c3]) successfully deleted
I20260925 00:53:22.843331 18906 catalog_manager.cc:4841] TS 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441): tablet 686171cea65c4394a50a83ed6387c0ad (table test-table [id=6bcf9524fffa47be9d96a424c89c43c3]) successfully deleted
I20260925 00:53:22.845288 19506 ts_tablet_manager.cc:1905] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:22.847718 19506 ts_tablet_manager.cc:1918] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:53:22.847792 19506 log.cc:1196] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/686171cea65c4394a50a83ed6387c0ad
I20260925 00:53:22.847983 19506 ts_tablet_manager.cc:1939] T 686171cea65c4394a50a83ed6387c0ad P aeef965e9b0443c88642d36c09c6be7a: Deleting consensus metadata
I20260925 00:53:22.848835 18907 catalog_manager.cc:4841] TS aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243): tablet 686171cea65c4394a50a83ed6387c0ad (table test-table [id=6bcf9524fffa47be9d96a424c89c43c3]) successfully deleted
Sep 25 00:53:22 dist-test-slave-rpk1 krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:53:22 dist-test-slave-rpk1 krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:22.871225 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:22.858975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.16.202.62:46661)
0925 00:53:22.859279 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:22.859288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:22.859318 (+ 30us) server_negotiation.cc:407] Connection header received
0925 00:53:22.859485 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:22.859488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:22.859537 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:22.859612 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:22.860420 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:22.861110 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:22.861960 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:22.862192 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:22.862280 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:22.868027 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:22.868030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:22.868032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:22.868071 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:22.869490 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:22.869909 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:22.869913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:22.869915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:22.870005 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:22.870269 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:22.870272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:22.870274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:22.870521 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:22.870662 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:22.870914 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 00:53:22.871050 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
I20260925 00:53:22.873993 18922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:22.874435 18922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:22.879343 19199 tablet_service.cc:1467] Processing CreateTablet for tablet 0372edfa4b564489a481c600da46847e (DEFAULT_TABLE table=test-table [id=255e66f185794de683deb93e8dc6209c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:22.879300 19059 tablet_service.cc:1467] Processing CreateTablet for tablet 0372edfa4b564489a481c600da46847e (DEFAULT_TABLE table=test-table [id=255e66f185794de683deb93e8dc6209c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:22.879405 19339 tablet_service.cc:1467] Processing CreateTablet for tablet 0372edfa4b564489a481c600da46847e (DEFAULT_TABLE table=test-table [id=255e66f185794de683deb93e8dc6209c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:22.879663 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0372edfa4b564489a481c600da46847e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:22.879663 19339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0372edfa4b564489a481c600da46847e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:22.879668 19059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0372edfa4b564489a481c600da46847e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:22.882128 19519 tablet_bootstrap.cc:492] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Bootstrap starting.
I20260925 00:53:22.882294 19518 tablet_bootstrap.cc:492] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Bootstrap starting.
I20260925 00:53:22.882548 19517 tablet_bootstrap.cc:492] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Bootstrap starting.
I20260925 00:53:22.883015 19519 tablet_bootstrap.cc:654] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:22.883158 19518 tablet_bootstrap.cc:654] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:22.883456 19517 tablet_bootstrap.cc:654] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:22.884215 19518 tablet_bootstrap.cc:492] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: No bootstrap required, opened a new log
I20260925 00:53:22.884282 19519 tablet_bootstrap.cc:492] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: No bootstrap required, opened a new log
I20260925 00:53:22.884330 19518 ts_tablet_manager.cc:1397] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:22.884373 19519 ts_tablet_manager.cc:1397] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:22.884775 19519 raft_consensus.cc:348] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.884776 19518 raft_consensus.cc:348] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.884902 19519 raft_consensus.cc:374] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:22.884927 19518 raft_consensus.cc:374] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:22.884956 19519 raft_consensus.cc:729] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Initialized, Role: FOLLOWER
I20260925 00:53:22.884966 19518 raft_consensus.cc:729] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Initialized, Role: FOLLOWER
I20260925 00:53:22.885079 19519 consensus_queue.cc:260] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.885088 19518 consensus_queue.cc:260] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.885354 19519 ts_tablet_manager.cc:1428] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:22.885357 19518 ts_tablet_manager.cc:1428] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:53:22.885143 19517 tablet_bootstrap.cc:492] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: No bootstrap required, opened a new log
I20260925 00:53:22.885808 19517 ts_tablet_manager.cc:1397] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:22.886483 19517 raft_consensus.cc:348] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.886602 19517 raft_consensus.cc:374] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:22.886644 19517 raft_consensus.cc:729] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeef965e9b0443c88642d36c09c6be7a, State: Initialized, Role: FOLLOWER
I20260925 00:53:22.886806 19517 consensus_queue.cc:260] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.887252 19517 ts_tablet_manager.cc:1428] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:22.945130 19501 raft_consensus.cc:482] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:22.945289 19501 raft_consensus.cc:504] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.945712 19501 leader_election.cc:288] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599)
I20260925 00:53:22.946251 19079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0372edfa4b564489a481c600da46847e" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a" is_pre_election: true
I20260925 00:53:22.946311 19219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0372edfa4b564489a481c600da46847e" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c598e2ffed94363a02d0d9cf8b56594" is_pre_election: true
I20260925 00:53:22.946455 19219 raft_consensus.cc:2454] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 0.
I20260925 00:53:22.946453 19079 raft_consensus.cc:2454] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 0.
I20260925 00:53:22.946800 19291 leader_election.cc:302] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793b90c55d514ded9313fd34884030f3, aeef965e9b0443c88642d36c09c6be7a; no voters:
I20260925 00:53:22.946995 19501 raft_consensus.cc:2781] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:22.947052 19501 raft_consensus.cc:482] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:22.947080 19501 raft_consensus.cc:3037] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:22.948055 19501 raft_consensus.cc:504] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.948379 19501 leader_election.cc:288] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 election: Requested vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599)
I20260925 00:53:22.948799 19079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0372edfa4b564489a481c600da46847e" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a"
I20260925 00:53:22.948810 19219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0372edfa4b564489a481c600da46847e" candidate_uuid: "793b90c55d514ded9313fd34884030f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c598e2ffed94363a02d0d9cf8b56594"
I20260925 00:53:22.948906 19079 raft_consensus.cc:3037] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:22.948920 19219 raft_consensus.cc:3037] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:22.949813 19079 raft_consensus.cc:2454] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 1.
I20260925 00:53:22.949817 19219 raft_consensus.cc:2454] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793b90c55d514ded9313fd34884030f3 in term 1.
I20260925 00:53:22.950064 19291 leader_election.cc:302] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, 793b90c55d514ded9313fd34884030f3; no voters:
I20260925 00:53:22.950227 19501 raft_consensus.cc:2781] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:22.950330 19501 raft_consensus.cc:686] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 1 LEADER]: Becoming Leader. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Running, Role: LEADER
I20260925 00:53:22.950469 19501 consensus_queue.cc:237] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:22.952198 18922 catalog_manager.cc:5495] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793b90c55d514ded9313fd34884030f3 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "793b90c55d514ded9313fd34884030f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } health_report { overall_health: HEALTHY } } }
I20260925 00:53:22.974146 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:22.970677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.16.202.3:46441)
0925 00:53:22.970968 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:22.970981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:22.970994 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:22.971035 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:22.971037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:22.971078 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:22.971146 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:22.971404 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:22.971934 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:22.972976 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:22.973231 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:22.973326 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:22.973592 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:22.973676 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:22.973736 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:22.973850 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 00:53:22.973981 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
I20260925 00:53:22.980625 19079 raft_consensus.cc:1261] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Refusing update from remote peer 793b90c55d514ded9313fd34884030f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:22.980718 19219 raft_consensus.cc:1261] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Refusing update from remote peer 793b90c55d514ded9313fd34884030f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:22.981056 19499 consensus_queue.cc:1035] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:22.981230 19501 consensus_queue.cc:1035] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:23.009476 18922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:23.009630 18922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:23.011683 18922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Sending DeleteTablet for 3 replicas of tablet 0372edfa4b564489a481c600da46847e
I20260925 00:53:23.012377 19339 tablet_service.cc:1514] Processing DeleteTablet for tablet 0372edfa4b564489a481c600da46847e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:23 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:57256
I20260925 00:53:23.012415 19199 tablet_service.cc:1514] Processing DeleteTablet for tablet 0372edfa4b564489a481c600da46847e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:23 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33734
I20260925 00:53:23.012444 19059 tablet_service.cc:1514] Processing DeleteTablet for tablet 0372edfa4b564489a481c600da46847e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:23 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:40700
I20260925 00:53:23.012573 19508 tablet_replica.cc:330] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: stopping tablet replica
I20260925 00:53:23.012632 19506 tablet_replica.cc:330] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: stopping tablet replica
I20260925 00:53:23.012727 19508 raft_consensus.cc:2229] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:23.012814 19506 raft_consensus.cc:2229] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:23.012934 19508 raft_consensus.cc:2258] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:23.012953 19506 raft_consensus.cc:2258] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:23.013160 19507 tablet_replica.cc:330] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: stopping tablet replica
I20260925 00:53:23.013301 19507 raft_consensus.cc:2229] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:23.013432 19507 raft_consensus.cc:2258] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:23.013775 19508 ts_tablet_manager.cc:1905] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:23.013901 18922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:53:23.014088 19507 ts_tablet_manager.cc:1905] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:23.014284 19506 ts_tablet_manager.cc:1905] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:53:23.014398 18922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:23.016742 19507 ts_tablet_manager.cc:1918] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:23.016819 19507 log.cc:1196] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/0372edfa4b564489a481c600da46847e
I20260925 00:53:23.016824 19508 ts_tablet_manager.cc:1918] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:23.016824 19506 ts_tablet_manager.cc:1918] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:23.016901 19508 log.cc:1196] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/0372edfa4b564489a481c600da46847e
I20260925 00:53:23.016901 19506 log.cc:1196] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/0372edfa4b564489a481c600da46847e
I20260925 00:53:23.017071 19507 ts_tablet_manager.cc:1939] T 0372edfa4b564489a481c600da46847e P 0c598e2ffed94363a02d0d9cf8b56594: Deleting consensus metadata
I20260925 00:53:23.017184 19506 ts_tablet_manager.cc:1939] T 0372edfa4b564489a481c600da46847e P aeef965e9b0443c88642d36c09c6be7a: Deleting consensus metadata
I20260925 00:53:23.017262 19508 ts_tablet_manager.cc:1939] T 0372edfa4b564489a481c600da46847e P 793b90c55d514ded9313fd34884030f3: Deleting consensus metadata
I20260925 00:53:23.017838 18907 catalog_manager.cc:4841] TS 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599): tablet 0372edfa4b564489a481c600da46847e (table test-table [id=255e66f185794de683deb93e8dc6209c]) successfully deleted
I20260925 00:53:23.018206 18907 catalog_manager.cc:4841] TS aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243): tablet 0372edfa4b564489a481c600da46847e (table test-table [id=255e66f185794de683deb93e8dc6209c]) successfully deleted
I20260925 00:53:23.018254 18906 catalog_manager.cc:4841] TS 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441): tablet 0372edfa4b564489a481c600da46847e (table test-table [id=255e66f185794de683deb93e8dc6209c]) successfully deleted
I20260925 00:53:23.019917 19059 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8edf67dace446a9fcbe1afb49ec6ba (DEFAULT_TABLE table=test-table [id=8a30197defb44e62801114931c764fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:23.019951 19199 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8edf67dace446a9fcbe1afb49ec6ba (DEFAULT_TABLE table=test-table [id=8a30197defb44e62801114931c764fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:23.019991 19339 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8edf67dace446a9fcbe1afb49ec6ba (DEFAULT_TABLE table=test-table [id=8a30197defb44e62801114931c764fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:23.020190 19059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8edf67dace446a9fcbe1afb49ec6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:23.020206 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8edf67dace446a9fcbe1afb49ec6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:23.020196 19339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8edf67dace446a9fcbe1afb49ec6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:23.022315 19517 tablet_bootstrap.cc:492] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a: Bootstrap starting.
I20260925 00:53:23.022466 19519 tablet_bootstrap.cc:492] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Bootstrap starting.
I20260925 00:53:23.022967 19518 tablet_bootstrap.cc:492] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Bootstrap starting.
I20260925 00:53:23.023152 19517 tablet_bootstrap.cc:654] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:23.023236 19519 tablet_bootstrap.cc:654] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:23.023711 19518 tablet_bootstrap.cc:654] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:23.024363 19517 tablet_bootstrap.cc:492] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a: No bootstrap required, opened a new log
I20260925 00:53:23.024456 19517 ts_tablet_manager.cc:1397] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:23.024735 19518 tablet_bootstrap.cc:492] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: No bootstrap required, opened a new log
I20260925 00:53:23.024822 19518 ts_tablet_manager.cc:1397] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:23.024935 19517 raft_consensus.cc:348] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.025053 19517 raft_consensus.cc:374] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:23.025089 19517 raft_consensus.cc:729] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeef965e9b0443c88642d36c09c6be7a, State: Initialized, Role: FOLLOWER
I20260925 00:53:23.025137 19519 tablet_bootstrap.cc:492] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: No bootstrap required, opened a new log
I20260925 00:53:23.025236 19519 ts_tablet_manager.cc:1397] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 00:53:23.025214 19517 consensus_queue.cc:260] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.025236 19518 raft_consensus.cc:348] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.025341 19518 raft_consensus.cc:374] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:23.025372 19518 raft_consensus.cc:729] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793b90c55d514ded9313fd34884030f3, State: Initialized, Role: FOLLOWER
I20260925 00:53:23.025489 19517 ts_tablet_manager.cc:1428] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:23.025493 19518 consensus_queue.cc:260] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.025615 19519 raft_consensus.cc:348] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.025731 19519 raft_consensus.cc:374] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:23.025745 19518 ts_tablet_manager.cc:1428] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:23.025769 19519 raft_consensus.cc:729] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Initialized, Role: FOLLOWER
I20260925 00:53:23.025887 19519 consensus_queue.cc:260] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.026145 19519 ts_tablet_manager.cc:1428] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:23.331422 19436 raft_consensus.cc:482] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:23.331562 19436 raft_consensus.cc:504] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.331995 19436 leader_election.cc:288] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441)
I20260925 00:53:23.332456 19079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8edf67dace446a9fcbe1afb49ec6ba" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a" is_pre_election: true
I20260925 00:53:23.332466 19359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8edf67dace446a9fcbe1afb49ec6ba" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793b90c55d514ded9313fd34884030f3" is_pre_election: true
I20260925 00:53:23.332603 19079 raft_consensus.cc:2454] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 0.
I20260925 00:53:23.332621 19359 raft_consensus.cc:2454] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 0.
I20260925 00:53:23.332954 19151 leader_election.cc:302] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, aeef965e9b0443c88642d36c09c6be7a; no voters:
I20260925 00:53:23.333137 19436 raft_consensus.cc:2781] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:23.333184 19436 raft_consensus.cc:482] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:23.333217 19436 raft_consensus.cc:3037] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:23.334378 19436 raft_consensus.cc:504] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.334756 19436 leader_election.cc:288] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 election: Requested vote from peers aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243), 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441)
I20260925 00:53:23.335101 19079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8edf67dace446a9fcbe1afb49ec6ba" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeef965e9b0443c88642d36c09c6be7a"
I20260925 00:53:23.335106 19359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8edf67dace446a9fcbe1afb49ec6ba" candidate_uuid: "0c598e2ffed94363a02d0d9cf8b56594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793b90c55d514ded9313fd34884030f3"
I20260925 00:53:23.335223 19359 raft_consensus.cc:3037] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:23.335237 19079 raft_consensus.cc:3037] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:23.336113 19079 raft_consensus.cc:2454] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 1.
I20260925 00:53:23.336141 19359 raft_consensus.cc:2454] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c598e2ffed94363a02d0d9cf8b56594 in term 1.
I20260925 00:53:23.336409 19151 leader_election.cc:302] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c598e2ffed94363a02d0d9cf8b56594, aeef965e9b0443c88642d36c09c6be7a; no voters:
I20260925 00:53:23.336560 19436 raft_consensus.cc:2781] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:23.336642 19436 raft_consensus.cc:686] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 LEADER]: Becoming Leader. State: Replica: 0c598e2ffed94363a02d0d9cf8b56594, State: Running, Role: LEADER
I20260925 00:53:23.336819 19436 consensus_queue.cc:237] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } }
I20260925 00:53:23.338390 18921 catalog_manager.cc:5495] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2). New cstate: current_term: 1 leader_uuid: "0c598e2ffed94363a02d0d9cf8b56594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c598e2ffed94363a02d0d9cf8b56594" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:23.399879 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:23.397039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.16.202.2:37599)
0925 00:53:23.397301 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:23.397307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:23.397318 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:53:23.397353 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:23.397356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:23.397396 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:23.397464 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:23.397671 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:23.398205 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:23.399006 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:23.399221 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:23.399307 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:23.399497 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:23.399562 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:23.399610 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:23.399683 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 00:53:23.399723 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":103,"threads_started":1}
I20260925 00:53:23.408385 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:23.405712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:58409 (local address 127.16.202.2:37599)
0925 00:53:23.405811 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:23.405814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:23.405952 (+ 138us) server_negotiation.cc:407] Connection header received
0925 00:53:23.406044 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:23.406046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:23.406084 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:23.406143 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:23.406317 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:23.406799 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:23.407563 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:23.407778 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:23.407852 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:23.408045 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:23.408103 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:23.408150 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:23.408211 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 00:53:23.408250 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:53:23.408396 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:23.405616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37599 (local address 127.16.202.2:58409)
0925 00:53:23.405880 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:23.405892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:23.405976 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:23.406155 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:23.406157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:23.406162 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:23.406248 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:23.406252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:23.406870 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:23.406872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:23.407468 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:23.407476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:23.407870 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:23.407873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:23.407939 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:23.407963 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:23.408160 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:23.408163 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:53:23.408202 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 00:53:23.408272 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260925 00:53:23.413676 19359 raft_consensus.cc:1261] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Refusing update from remote peer 0c598e2ffed94363a02d0d9cf8b56594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:23.413815 19079 raft_consensus.cc:1261] T 6b8edf67dace446a9fcbe1afb49ec6ba P aeef965e9b0443c88642d36c09c6be7a [term 1 FOLLOWER]: Refusing update from remote peer 0c598e2ffed94363a02d0d9cf8b56594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:23.414070 19486 consensus_queue.cc:1035] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793b90c55d514ded9313fd34884030f3" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:23.414212 19436 consensus_queue.cc:1035] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeef965e9b0443c88642d36c09c6be7a" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:53:23.427647 19484 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 00:53:24.983436 18920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:24.983762 18920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:24.986606 18920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d982a314a894083a902fc3cbaf3da8d: Sending DeleteTablet for 3 replicas of tablet 6b8edf67dace446a9fcbe1afb49ec6ba
I20260925 00:53:24.987285 19199 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8edf67dace446a9fcbe1afb49ec6ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:24 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33734
I20260925 00:53:24.987387 19059 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8edf67dace446a9fcbe1afb49ec6ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:24 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:40700
I20260925 00:53:24.987697 19339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8edf67dace446a9fcbe1afb49ec6ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:24 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:57256
I20260925 00:53:24.988045 19562 tablet_replica.cc:330] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: stopping tablet replica
I20260925 00:53:24.988045 19560 tablet_replica.cc:330] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: stopping tablet replica
I20260925 00:53:24.988183 19562 raft_consensus.cc:2229] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:24.988265 19560 raft_consensus.cc:2229] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:24.988343 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18994
I20260925 00:53:24.989482 19560 raft_consensus.cc:2258] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:24.990267 19560 ts_tablet_manager.cc:1905] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:24.992988 19562 raft_consensus.cc:2258] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:24.993239 19560 ts_tablet_manager.cc:1918] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:53:24.993372 19560 log.cc:1196] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/6b8edf67dace446a9fcbe1afb49ec6ba
I20260925 00:53:24.993700 19560 ts_tablet_manager.cc:1939] T 6b8edf67dace446a9fcbe1afb49ec6ba P 0c598e2ffed94363a02d0d9cf8b56594: Deleting consensus metadata
I20260925 00:53:24.993764 19562 ts_tablet_manager.cc:1905] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:24.994650 18907 catalog_manager.cc:4841] TS 0c598e2ffed94363a02d0d9cf8b56594 (127.16.202.2:37599): tablet 6b8edf67dace446a9fcbe1afb49ec6ba (table test-table [id=8a30197defb44e62801114931c764fe1]) successfully deleted
W20260925 00:53:24.996006 18907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:34243 (error 108)
I20260925 00:53:24.996073 19562 ts_tablet_manager.cc:1918] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:53:24.996135 19562 log.cc:1196] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/6b8edf67dace446a9fcbe1afb49ec6ba
I20260925 00:53:24.996371 19562 ts_tablet_manager.cc:1939] T 6b8edf67dace446a9fcbe1afb49ec6ba P 793b90c55d514ded9313fd34884030f3: Deleting consensus metadata
I20260925 00:53:24.997119 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:24.996728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:34243 (local address 127.0.0.1:53232)
0925 00:53:24.996944 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:24.997018 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260925 00:53:24.997187 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 19134
I20260925 00:53:24.997160 18906 catalog_manager.cc:4841] TS 793b90c55d514ded9313fd34884030f3 (127.16.202.3:46441): tablet 6b8edf67dace446a9fcbe1afb49ec6ba (table test-table [id=8a30197defb44e62801114931c764fe1]) successfully deleted
W20260925 00:53:24.997918 18907 catalog_manager.cc:4584] TS aeef965e9b0443c88642d36c09c6be7a (127.16.202.1:34243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b8edf67dace446a9fcbe1afb49ec6ba: Network error: Client connection negotiation failed: client connection to 127.16.202.1:34243: connect: Connection refused (error 111)
I20260925 00:53:25.003937 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 19274
I20260925 00:53:25.010380 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 18890
2026-09-25T00:53:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6516 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:25 dist-test-slave-rpk1 krb5kdc[19567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:25 dist-test-slave-rpk1 krb5kdc[19567](info): set up 2 sockets
Sep 25 00:53:25 dist-test-slave-rpk1 krb5kdc[19567](info): commencing operation
krb5kdc: starting...
W20260925 00:53:25.977126 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:26.100565 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34111
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:26.212507 19580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:26.212852 19580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:26.212913 19580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:26.217046 19580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:26.217116 19580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:26.217142 19580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:26.217162 19580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:26.217182 19580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:26.222945 19580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:26.224121 19580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:26.225247 19580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:26.234751 19580 server_base.cc:1034] running on GCE node
W20260925 00:53:26.234673 19588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.234673 19586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.234683 19585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:26.235656 19580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:26.237021 19580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:26.238224 19580 hybrid_clock.cc:648] HybridClock initialized: now 1790297606238199 us; error 40 us; skew 500 ppm
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:26.241261 19580 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:26.242240 19580 webserver.cc:460] Webserver started at http://127.16.202.62:37421/ using document root <none> and password file <none>
I20260925 00:53:26.242805 19580 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:26.242874 19580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:26.243119 19580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:26.245326 19580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "cb292f449eab40f996d307d768f79554"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "8d8e445ab80e75e8811f7540c88e96dd"
server_key_iv: "8f3713b6cf7b34134c579cab3bf22e93"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.245918 19580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "cb292f449eab40f996d307d768f79554"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "8d8e445ab80e75e8811f7540c88e96dd"
server_key_iv: "8f3713b6cf7b34134c579cab3bf22e93"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.249828 19580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:26.252310 19595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.253657 19580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:26.253795 19580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "cb292f449eab40f996d307d768f79554"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "8d8e445ab80e75e8811f7540c88e96dd"
server_key_iv: "8f3713b6cf7b34134c579cab3bf22e93"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.253926 19580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:26.265918 19580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:26.269378 19580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:26.269574 19580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:26.276604 19580 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:34111
I20260925 00:53:26.276757 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:34111 every 8 connection(s)
I20260925 00:53:26.277668 19580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:26.280589 19648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:26.286586 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554: Bootstrap starting.
I20260925 00:53:26.287393 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 19580
I20260925 00:53:26.287495 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:26.287752 17192 external_mini_cluster.cc:1428] Setting key a7a46e7092245fc2ab355f6ae2a4bcf7
I20260925 00:53:26.289541 19648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:26.290512 19648 log.cc:824] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:26.292848 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554: No bootstrap required, opened a new log
I20260925 00:53:26.297187 19648 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb292f449eab40f996d307d768f79554" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34111 } }
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:26.297540 19648 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:26.297608 19648 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb292f449eab40f996d307d768f79554, State: Initialized, Role: FOLLOWER
I20260925 00:53:26.298090 19648 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb292f449eab40f996d307d768f79554" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34111 } }
I20260925 00:53:26.298249 19648 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:26.298321 19648 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:26.298444 19648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:26.299716 19648 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb292f449eab40f996d307d768f79554" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34111 } }
I20260925 00:53:26.300143 19648 leader_election.cc:302] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb292f449eab40f996d307d768f79554; no voters:
I20260925 00:53:26.300540 19648 leader_election.cc:288] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:26.300716 19653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:26.300976 19653 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [term 1 LEADER]: Becoming Leader. State: Replica: cb292f449eab40f996d307d768f79554, State: Running, Role: LEADER
I20260925 00:53:26.301348 19653 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb292f449eab40f996d307d768f79554" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34111 } }
I20260925 00:53:26.301898 19648 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:26.302729 19653 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb292f449eab40f996d307d768f79554. Latest consensus state: current_term: 1 leader_uuid: "cb292f449eab40f996d307d768f79554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb292f449eab40f996d307d768f79554" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34111 } } }
I20260925 00:53:26.302848 19653 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:26.303144 19654 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb292f449eab40f996d307d768f79554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb292f449eab40f996d307d768f79554" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34111 } } }
I20260925 00:53:26.303236 19654 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:26.303284 19662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:26.304196 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:26.289058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.16.202.62:34111)
0925 00:53:26.289544 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:26.289564 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:26.289621 (+ 57us) server_negotiation.cc:407] Connection header received
0925 00:53:26.290534 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:26.290562 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:26.291021 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:26.291340 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:26.292081 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.293580 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.294514 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.295264 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.295452 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:26.298244 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:26.298251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:26.298270 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:26.298312 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:26.300330 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:26.300993 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:26.301002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:26.301010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:26.301171 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:26.301537 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:26.301540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:26.301542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:26.302100 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:26.302284 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:26.302762 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 00:53:26.303113 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":126,"threads_started":1}
I20260925 00:53:26.306864 19662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:26.313076 19662 catalog_manager.cc:1347] Generated new cluster ID: e554f526bdc649bf9267d848ff126e4b
I20260925 00:53:26.313169 19662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:26.324028 19662 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:26.325181 19662 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:26.330801 19662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb292f449eab40f996d307d768f79554: Generated new TSK 0
I20260925 00:53:26.331600 19662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:26.365548 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34111
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:26.513367 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:26.513648 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:26.513690 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:26.517690 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:26.517758 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:26.517819 19674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:26.523128 19674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:26.524163 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:26.525239 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:26.534952 19680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.534952 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.535048 19682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:26.535405 19674 server_base.cc:1034] running on GCE node
I20260925 00:53:26.535827 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:26.536484 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:26.537679 19674 hybrid_clock.cc:648] HybridClock initialized: now 1790297606537665 us; error 54 us; skew 500 ppm
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:26.540700 19674 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:26.541669 19674 webserver.cc:460] Webserver started at http://127.16.202.1:37511/ using document root <none> and password file <none>
I20260925 00:53:26.542222 19674 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:26.542289 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:26.542492 19674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:26.544579 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "32c4945da0674d0581672c603546792a"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "8fdd59289605a205d87fc108f544e46a"
server_key_iv: "bdd56e9f1f185b738b555a756de6da5f"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.545182 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "32c4945da0674d0581672c603546792a"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "8fdd59289605a205d87fc108f544e46a"
server_key_iv: "bdd56e9f1f185b738b555a756de6da5f"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.548929 19674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:53:26.551402 19689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.552835 19674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:26.552949 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "32c4945da0674d0581672c603546792a"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "8fdd59289605a205d87fc108f544e46a"
server_key_iv: "bdd56e9f1f185b738b555a756de6da5f"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.553067 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:26.569453 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:26.573117 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:26.573277 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:26.573839 19674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:26.574643 19674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:26.574690 19674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.574729 19674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:26.574757 19674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.583132 19674 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:35195
I20260925 00:53:26.583173 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:35195 every 8 connection(s)
I20260925 00:53:26.584013 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:26.593520 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 19674
I20260925 00:53:26.593617 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:26.593883 17192 external_mini_cluster.cc:1428] Setting key a5f77302bc2f882ff255eb22df6ece40
I20260925 00:53:26.597855 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:26.585798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:57589 (local address 127.16.202.62:34111)
0925 00:53:26.585936 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:26.585940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:26.586808 (+ 868us) server_negotiation.cc:407] Connection header received
0925 00:53:26.588057 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:26.588059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:26.588105 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:26.588174 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:26.589434 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.589970 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.591212 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.591452 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.591532 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:26.593952 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:26.593956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:26.593959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:26.593987 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:26.595674 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:26.596171 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:26.596175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:26.596177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:26.596265 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:26.596589 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:26.596592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:26.596593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:26.596836 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:26.596928 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:26.597608 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 00:53:26.597732 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:26.598980 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:26.586210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34111 (local address 127.16.202.1:57589)
0925 00:53:26.586655 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:26.586698 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:26.587868 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:26.588361 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:26.588367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:26.588784 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:26.589275 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:26.589290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.590100 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.590103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:26.591093 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:26.591103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.591575 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.591578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:26.591718 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:26.592259 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:26.592275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:26.593778 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:26.595794 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:26.595799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:26.595813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:26.596080 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:26.596340 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:26.596342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:26.596343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:26.596512 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:26.596953 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:26.596961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:26.597306 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 00:53:26.597565 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 00:53:26.597932 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260925 00:53:26.600301 19803 heartbeater.cc:347] Connected to a master server at 127.16.202.62:34111
I20260925 00:53:26.600559 19803 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:26.601125 19803 heartbeater.cc:511] Master 127.16.202.62:34111 requested a full tablet report, sending...
I20260925 00:53:26.602773 19609 ts_manager.cc:194] Registered new tserver with Master: 32c4945da0674d0581672c603546792a (127.16.202.1:35195)
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:26.603911 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:57589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:26.619733 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34111
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:26.731385 19808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:26.731709 19808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:26.731776 19808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:26.735931 19808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:26.736001 19808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:26.736080 19808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:26.741498 19808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:26.742563 19808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:26.743686 19808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:26.753206 19816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.753207 19813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.753226 19814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:26.753572 19808 server_base.cc:1034] running on GCE node
I20260925 00:53:26.753969 19808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:26.754599 19808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:26.755769 19808 hybrid_clock.cc:648] HybridClock initialized: now 1790297606755726 us; error 50 us; skew 500 ppm
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:26.758687 19808 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:26.759577 19808 webserver.cc:460] Webserver started at http://127.16.202.2:37923/ using document root <none> and password file <none>
I20260925 00:53:26.760111 19808 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:26.760176 19808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:26.760411 19808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:26.762569 19808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "b6f6e20fd0de4dbb8fec91dadc3b7bb0"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "e1847318281bbb8747aebc4f67b86de4"
server_key_iv: "85aa21bffa8076b963e81b8084591bb4"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.763111 19808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "b6f6e20fd0de4dbb8fec91dadc3b7bb0"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "e1847318281bbb8747aebc4f67b86de4"
server_key_iv: "85aa21bffa8076b963e81b8084591bb4"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.767030 19808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:26.769606 19823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.770939 19808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:26.771066 19808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "b6f6e20fd0de4dbb8fec91dadc3b7bb0"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "e1847318281bbb8747aebc4f67b86de4"
server_key_iv: "85aa21bffa8076b963e81b8084591bb4"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.771189 19808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:26.784843 19808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:26.788477 19808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:26.788721 19808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:26.789381 19808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:26.790378 19808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:26.790434 19808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.790494 19808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:26.790529 19808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.799330 19808 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:34421
I20260925 00:53:26.799386 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:34421 every 8 connection(s)
I20260925 00:53:26.800269 19808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:26.806152 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 19808
I20260925 00:53:26.806272 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:26.806568 17192 external_mini_cluster.cc:1428] Setting key cbae5932023191ad6d8496654d9247ce
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:26.815225 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:26.802111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:57705 (local address 127.16.202.62:34111)
0925 00:53:26.802238 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:26.802242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:26.803248 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 00:53:26.804752 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:26.804756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:26.804820 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:26.804937 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:26.806174 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.806909 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.808306 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.808596 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.808734 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:26.811313 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:26.811317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:26.811319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:26.811348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:26.812989 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:26.813489 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:26.813493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:26.813495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:26.813591 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:26.813930 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:26.813934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:26.813936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:26.814189 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:26.814293 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:26.814918 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 00:53:26.815044 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:26.816200 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:26.802574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34111 (local address 127.16.202.2:57705)
0925 00:53:26.803081 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:26.803141 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:26.804462 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:26.805099 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:26.805106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:26.805515 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:26.806011 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:26.806025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.807044 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.807048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:26.808183 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:26.808193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:26.808775 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:26.808778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:26.808944 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:26.809523 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:26.809546 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:26.811145 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:26.813115 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:26.813120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:26.813134 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:26.813394 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:26.813686 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:26.813688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:26.813690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:26.813833 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:26.814344 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:26.814350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:26.814657 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 00:53:26.814842 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 00:53:26.815135 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260925 00:53:26.817605 19937 heartbeater.cc:347] Connected to a master server at 127.16.202.62:34111
I20260925 00:53:26.817930 19937 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:26.818516 19937 heartbeater.cc:511] Master 127.16.202.62:34111 requested a full tablet report, sending...
I20260925 00:53:26.819798 19609 ts_manager.cc:194] Registered new tserver with Master: b6f6e20fd0de4dbb8fec91dadc3b7bb0 (127.16.202.2:34421)
I20260925 00:53:26.820402 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:57705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:26.836153 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34111
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:26.948424 19942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:26.948760 19942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:26.948823 19942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:26.953519 19942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:26.953596 19942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:26.953683 19942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:26.959350 19942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:26.960530 19942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:26.961647 19942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:26.971685 19950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.971657 19947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:26.971700 19948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:26.971674 19942 server_base.cc:1034] running on GCE node
I20260925 00:53:26.972345 19942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:26.973083 19942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:26.974279 19942 hybrid_clock.cc:648] HybridClock initialized: now 1790297606974257 us; error 33 us; skew 500 ppm
Sep 25 00:53:26 dist-test-slave-rpk1 krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:26.977242 19942 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:26.978139 19942 webserver.cc:460] Webserver started at http://127.16.202.3:44301/ using document root <none> and password file <none>
I20260925 00:53:26.978672 19942 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:26.978730 19942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:26.978911 19942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:26.981031 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "4d4079f08cf549dbaf6b5c50bb098580"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "e76d22c57c8b4c7e6a4cc872eb9bc540"
server_key_iv: "fb46954e08f1f580d5573df4e1267e3c"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.981534 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "4d4079f08cf549dbaf6b5c50bb098580"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "e76d22c57c8b4c7e6a4cc872eb9bc540"
server_key_iv: "fb46954e08f1f580d5573df4e1267e3c"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.985351 19942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:53:26.988065 19957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:26.989535 19942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:26.989670 19942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "4d4079f08cf549dbaf6b5c50bb098580"
format_stamp: "Formatted at 2026-09-25 00:53:26 on dist-test-slave-rpk1"
server_key: "e76d22c57c8b4c7e6a4cc872eb9bc540"
server_key_iv: "fb46954e08f1f580d5573df4e1267e3c"
server_key_version: "encryptionkey@0"
I20260925 00:53:26.989786 19942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:27.005122 19942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:27.008297 19942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:27.008495 19942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:27.009140 19942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:27.010006 19942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:27.010056 19942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:27.010110 19942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:27.010136 19942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:27.018510 19942 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:35973
I20260925 00:53:27.018532 20070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:35973 every 8 connection(s)
I20260925 00:53:27.019425 19942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:27.022590 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 19942
I20260925 00:53:27.022706 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:27.022985 17192 external_mini_cluster.cc:1428] Setting key cd4708ef56a166544066e258c1b1ef6a
Sep 25 00:53:27 dist-test-slave-rpk1 krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:27.032647 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:27.021386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:49535 (local address 127.16.202.62:34111)
0925 00:53:27.021557 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:27.021561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:27.022391 (+ 830us) server_negotiation.cc:407] Connection header received
0925 00:53:27.023756 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:27.023759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:27.023813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:27.023884 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:27.025471 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:27.025987 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:27.027177 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:27.027434 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:27.027513 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:27.029761 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:27.029765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:27.029767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:27.029791 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:27.030932 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:27.031408 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:27.031410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:27.031411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:27.031472 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:27.031710 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:27.031712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:27.031713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:27.031866 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:27.031947 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:27.032423 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 00:53:27.032524 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:53:27.033473 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:27.021735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34111 (local address 127.16.202.3:49535)
0925 00:53:27.022220 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:27.022273 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:27.023544 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:27.024055 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:27.024063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:27.024616 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:27.025328 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:27.025345 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:27.026113 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:27.026116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:27.027063 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:27.027072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:27.027551 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:27.027554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:27.027691 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:27.028200 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:27.028215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:27.029614 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:27.031035 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:27.031040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:27.031052 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:27.031336 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:27.031531 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:27.031533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:27.031534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:27.031651 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:27.031945 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:27.031949 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:27.032182 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 00:53:27.032377 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 00:53:27.032624 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":100,"threads_started":1}
I20260925 00:53:27.034513 20071 heartbeater.cc:347] Connected to a master server at 127.16.202.62:34111
I20260925 00:53:27.034745 20071 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:27.035218 20071 heartbeater.cc:511] Master 127.16.202.62:34111 requested a full tablet report, sending...
I20260925 00:53:27.036309 19609 ts_manager.cc:194] Registered new tserver with Master: 4d4079f08cf549dbaf6b5c50bb098580 (127.16.202.3:35973)
I20260925 00:53:27.037117 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:49535
I20260925 00:53:27.037149 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 00:53:27.049237 19651 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:27.047887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.16.202.62:34111)
0925 00:53:27.048002 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:27.048006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:27.048155 (+ 149us) server_negotiation.cc:407] Connection header received
0925 00:53:27.048310 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:27.048312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:27.048357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:27.048429 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:27.049112 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34314: BlockingRecv error: recv got EOF from 127.0.0.1:34314 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:53:27.050258 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 19674
I20260925 00:53:27.055665 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 19808
I20260925 00:53:27.060169 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 19942
I20260925 00:53:27.065088 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 19580
2026-09-25T00:53:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2052 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:27 dist-test-slave-rpk1 krb5kdc[20083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:27 dist-test-slave-rpk1 krb5kdc[20083](info): set up 2 sockets
Sep 25 00:53:27 dist-test-slave-rpk1 krb5kdc[20083](info): commencing operation
krb5kdc: starting...
W20260925 00:53:28.024812 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:28 dist-test-slave-rpk1 krb5kdc[20083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:28.148041 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39073
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:28.261036 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:28.261417 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:28.261466 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:28.265635 20096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:28.265694 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:28.265713 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:28.265729 20096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:28.265743 20096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:28.271265 20096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:28.272354 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:28.273406 20096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:28.282184 20096 server_base.cc:1034] running on GCE node
W20260925 00:53:28.282138 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:28.282140 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:28.282138 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:28.282974 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:28.284202 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:28.285404 20096 hybrid_clock.cc:648] HybridClock initialized: now 1790297608285377 us; error 41 us; skew 500 ppm
Sep 25 00:53:28 dist-test-slave-rpk1 krb5kdc[20083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297608, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:28.291325 20096 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:28.292363 20096 webserver.cc:460] Webserver started at http://127.16.202.62:36497/ using document root <none> and password file <none>
I20260925 00:53:28.292989 20096 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:28.293061 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:28.293274 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:28.295414 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "3bc667152b5b4000a3ea2dc28653933e"
format_stamp: "Formatted at 2026-09-25 00:53:28 on dist-test-slave-rpk1"
server_key: "4e25b52c60081f6fd9f18a17df40b8e0"
server_key_iv: "a66f19b1b981c2f03bb90847715f401d"
server_key_version: "encryptionkey@0"
I20260925 00:53:28.296005 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "3bc667152b5b4000a3ea2dc28653933e"
format_stamp: "Formatted at 2026-09-25 00:53:28 on dist-test-slave-rpk1"
server_key: "4e25b52c60081f6fd9f18a17df40b8e0"
server_key_iv: "a66f19b1b981c2f03bb90847715f401d"
server_key_version: "encryptionkey@0"
I20260925 00:53:28.300020 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:28.302646 20111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:28.303923 20096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:28.304047 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "3bc667152b5b4000a3ea2dc28653933e"
format_stamp: "Formatted at 2026-09-25 00:53:28 on dist-test-slave-rpk1"
server_key: "4e25b52c60081f6fd9f18a17df40b8e0"
server_key_iv: "a66f19b1b981c2f03bb90847715f401d"
server_key_version: "encryptionkey@0"
I20260925 00:53:28.304154 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:28.316705 20096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:28.319543 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:28.319736 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:28.326855 20096 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39073
I20260925 00:53:28.327128 20163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39073 every 8 connection(s)
I20260925 00:53:28.327847 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:28.330922 20164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:28.334476 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20096
I20260925 00:53:28.334587 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:28.334841 17192 external_mini_cluster.cc:1428] Setting key 640f9f064a223545f3dba03df56a92ca
I20260925 00:53:28.336925 20164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e: Bootstrap starting.
I20260925 00:53:28.339663 20164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:28.340595 20164 log.cc:824] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:28.342864 20164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e: No bootstrap required, opened a new log
Sep 25 00:53:28 dist-test-slave-rpk1 krb5kdc[20083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:28.347054 20164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc667152b5b4000a3ea2dc28653933e" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39073 } }
I20260925 00:53:28.347329 20164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:28.347368 20164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc667152b5b4000a3ea2dc28653933e, State: Initialized, Role: FOLLOWER
I20260925 00:53:28.347864 20164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc667152b5b4000a3ea2dc28653933e" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39073 } }
I20260925 00:53:28.348006 20164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:28.348073 20164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:28.348186 20164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:28.349232 20164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc667152b5b4000a3ea2dc28653933e" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39073 } }
I20260925 00:53:28.349589 20164 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc667152b5b4000a3ea2dc28653933e; no voters:
I20260925 00:53:28.349927 20164 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:28.350021 20169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:28.350135 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:28.336272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.16.202.62:39073)
0925 00:53:28.336783 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:28.336803 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:28.336845 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:53:28.337657 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:28.337683 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:28.338131 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:28.338507 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:28.339207 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:28.340707 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:28.341595 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:28.342264 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:28.342423 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:28.344896 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:28.344903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:28.344922 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:28.344968 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:28.346980 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:28.347500 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:28.347505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:28.347510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:28.347630 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:28.347905 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:28.347908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:28.347909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:28.348412 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:28.348609 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:28.348921 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 00:53:28.349231 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260925 00:53:28.350193 20169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [term 1 LEADER]: Becoming Leader. State: Replica: 3bc667152b5b4000a3ea2dc28653933e, State: Running, Role: LEADER
I20260925 00:53:28.350574 20169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc667152b5b4000a3ea2dc28653933e" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39073 } }
I20260925 00:53:28.351141 20164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:28.352144 20171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bc667152b5b4000a3ea2dc28653933e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc667152b5b4000a3ea2dc28653933e" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39073 } } }
I20260925 00:53:28.352355 20171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:28.352694 20170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc667152b5b4000a3ea2dc28653933e. Latest consensus state: current_term: 1 leader_uuid: "3bc667152b5b4000a3ea2dc28653933e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc667152b5b4000a3ea2dc28653933e" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39073 } } }
I20260925 00:53:28.352847 20170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:28.353159 20178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:28.355059 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:53:28.356240 20178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:28.362650 20178 catalog_manager.cc:1347] Generated new cluster ID: 5411ff80ceb34e01800056ee5396accb
I20260925 00:53:28.362792 20178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:28.375916 20178 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:28.377084 20178 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:28.377627 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:28.371925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.16.202.62:39073)
0925 00:53:28.372042 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:28.372047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:28.372319 (+ 272us) server_negotiation.cc:407] Connection header received
0925 00:53:28.372438 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:28.372441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:28.372499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:28.372586 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:28.373180 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:28.374010 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:28.374935 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:28.375257 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:28.375355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:28.376493 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:28.376497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:28.376500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:28.377126 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:28.377162 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:28.377183 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:28.377281 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:28.377406 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 00:53:28.377487 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:53:28.382323 20178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc667152b5b4000a3ea2dc28653933e: Generated new TSK 0
I20260925 00:53:28.383181 20178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:53:28.388149 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20096
2026-09-25T00:53:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1320 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:28 dist-test-slave-rpk1 krb5kdc[20197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:28 dist-test-slave-rpk1 krb5kdc[20197](info): set up 2 sockets
Sep 25 00:53:28 dist-test-slave-rpk1 krb5kdc[20197](info): commencing operation
krb5kdc: starting...
W20260925 00:53:29.340618 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:29 dist-test-slave-rpk1 krb5kdc[20197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:29.463924 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33529
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:33529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:29.575410 20210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:29.575704 20210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:29.575760 20210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:29.579865 20210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:29.579939 20210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:29.579964 20210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:29.579986 20210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:29.580006 20210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:29.585649 20210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:33529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:29.586829 20210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:29.587925 20210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:29.598407 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:29.598322 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:29.598322 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:29.599655 20210 server_base.cc:1034] running on GCE node
I20260925 00:53:29.600391 20210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:29.601635 20210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:29.602835 20210 hybrid_clock.cc:648] HybridClock initialized: now 1790297609602813 us; error 32 us; skew 500 ppm
Sep 25 00:53:29 dist-test-slave-rpk1 krb5kdc[20197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:29.605792 20210 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:29.606812 20210 webserver.cc:460] Webserver started at http://127.16.202.62:46377/ using document root <none> and password file <none>
I20260925 00:53:29.607420 20210 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:29.607488 20210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:29.607681 20210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:29.609822 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "06a846c77f904f6994915957349bbcd7"
format_stamp: "Formatted at 2026-09-25 00:53:29 on dist-test-slave-rpk1"
server_key: "eee30642cb7a826f5fe6c388aed4442c"
server_key_iv: "3dd4423e51eec2b3f42f5e9272a376dd"
server_key_version: "encryptionkey@0"
I20260925 00:53:29.610361 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "06a846c77f904f6994915957349bbcd7"
format_stamp: "Formatted at 2026-09-25 00:53:29 on dist-test-slave-rpk1"
server_key: "eee30642cb7a826f5fe6c388aed4442c"
server_key_iv: "3dd4423e51eec2b3f42f5e9272a376dd"
server_key_version: "encryptionkey@0"
I20260925 00:53:29.614295 20210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:29.617009 20225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:29.618361 20210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:53:29.618492 20210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "06a846c77f904f6994915957349bbcd7"
format_stamp: "Formatted at 2026-09-25 00:53:29 on dist-test-slave-rpk1"
server_key: "eee30642cb7a826f5fe6c388aed4442c"
server_key_iv: "3dd4423e51eec2b3f42f5e9272a376dd"
server_key_version: "encryptionkey@0"
I20260925 00:53:29.618602 20210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:29.628115 20210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:29.631340 20210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:29.631544 20210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:29.638892 20210 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:33529
I20260925 00:53:29.639154 20277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:33529 every 8 connection(s)
I20260925 00:53:29.639966 20210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:29.640389 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20210
I20260925 00:53:29.640501 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:29.640828 17192 external_mini_cluster.cc:1428] Setting key c4c92c68e150a84575cce9a284fe6e06
I20260925 00:53:29.643286 20278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:29.650400 20278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7: Bootstrap starting.
Sep 25 00:53:29 dist-test-slave-rpk1 krb5kdc[20197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:29.653296 20278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:29.654245 20278 log.cc:824] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:29.656430 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:29.642325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.16.202.62:33529)
0925 00:53:29.642767 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:29.642789 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:29.642837 (+ 48us) server_negotiation.cc:407] Connection header received
0925 00:53:29.643737 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:29.643764 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:29.644214 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:29.644603 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:29.645346 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:29.646899 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:29.647805 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:29.648527 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:29.648756 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:29.651060 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:29.651067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:29.651086 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:29.651123 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:29.653166 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:29.653684 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:29.653692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:29.653698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:29.653850 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:29.654156 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:29.654160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:29.654162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:29.654727 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:29.654940 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:29.655276 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 00:53:29.655582 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260925 00:53:29.656497 20278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7: No bootstrap required, opened a new log
I20260925 00:53:29.660125 20278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a846c77f904f6994915957349bbcd7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33529 } }
I20260925 00:53:29.660420 20278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:29.660485 20278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a846c77f904f6994915957349bbcd7, State: Initialized, Role: FOLLOWER
I20260925 00:53:29.661033 20278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a846c77f904f6994915957349bbcd7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33529 } }
I20260925 00:53:29.661197 20278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:29.661291 20278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:29.661396 20278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:29.662571 20278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a846c77f904f6994915957349bbcd7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33529 } }
I20260925 00:53:29.662943 20278 leader_election.cc:302] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a846c77f904f6994915957349bbcd7; no voters:
I20260925 00:53:29.663340 20278 leader_election.cc:288] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:29.663455 20283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:29.663666 20283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [term 1 LEADER]: Becoming Leader. State: Replica: 06a846c77f904f6994915957349bbcd7, State: Running, Role: LEADER
I20260925 00:53:29.664039 20283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a846c77f904f6994915957349bbcd7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33529 } }
I20260925 00:53:29.664562 20278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:29.665660 20285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a846c77f904f6994915957349bbcd7. Latest consensus state: current_term: 1 leader_uuid: "06a846c77f904f6994915957349bbcd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a846c77f904f6994915957349bbcd7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33529 } } }
I20260925 00:53:29.665796 20285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:29.666111 20284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a846c77f904f6994915957349bbcd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a846c77f904f6994915957349bbcd7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33529 } } }
I20260925 00:53:29.666167 20292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:29.666263 20284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:29.669016 20292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:29.674480 20292 catalog_manager.cc:1347] Generated new cluster ID: 27151d70a2ea4ebbae6af37abc6af872
I20260925 00:53:29.674554 20292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:29.698444 20292 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:29.699762 20292 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:29.719251 20292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a846c77f904f6994915957349bbcd7: Generated new TSK 0
I20260925 00:53:29.720084 20292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:29.755983 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33529
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:29.884722 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:29.885059 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:29.885128 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:29.889489 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:29.889582 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:29.889672 20304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:29.895725 20304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:29.896955 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:29.898036 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:29.907836 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:29.907845 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:29.908006 20304 server_base.cc:1034] running on GCE node
W20260925 00:53:29.907850 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:29.908535 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:29.909230 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:29.910410 20304 hybrid_clock.cc:648] HybridClock initialized: now 1790297609910382 us; error 40 us; skew 500 ppm
Sep 25 00:53:29 dist-test-slave-rpk1 krb5kdc[20197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:29.913524 20304 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:29.914551 20304 webserver.cc:460] Webserver started at http://127.16.202.1:43165/ using document root <none> and password file <none>
I20260925 00:53:29.915124 20304 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:29.915187 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:29.915400 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:29.917543 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "6f6cdca219974e1abfb2bb16bc9ec43e"
format_stamp: "Formatted at 2026-09-25 00:53:29 on dist-test-slave-rpk1"
server_key: "0fcce73d5ab2a1415d47139945274e72"
server_key_iv: "5113b5cea4efacbacbe95ee64c4b81f1"
server_key_version: "encryptionkey@0"
I20260925 00:53:29.918085 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6cdca219974e1abfb2bb16bc9ec43e"
format_stamp: "Formatted at 2026-09-25 00:53:29 on dist-test-slave-rpk1"
server_key: "0fcce73d5ab2a1415d47139945274e72"
server_key_iv: "5113b5cea4efacbacbe95ee64c4b81f1"
server_key_version: "encryptionkey@0"
I20260925 00:53:29.921929 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 00:53:29.924583 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:29.925979 20304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:29.926092 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "6f6cdca219974e1abfb2bb16bc9ec43e"
format_stamp: "Formatted at 2026-09-25 00:53:29 on dist-test-slave-rpk1"
server_key: "0fcce73d5ab2a1415d47139945274e72"
server_key_iv: "5113b5cea4efacbacbe95ee64c4b81f1"
server_key_version: "encryptionkey@0"
I20260925 00:53:29.926211 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:29.944767 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:29.948004 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:29.948176 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:29.948808 20304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:29.949663 20304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:29.949710 20304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:29.949747 20304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:29.949774 20304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:29.958508 20304 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:34519
I20260925 00:53:29.958537 20432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:34519 every 8 connection(s)
I20260925 00:53:29.959461 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:29.964510 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20304
I20260925 00:53:29.964664 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:29.965045 17192 external_mini_cluster.cc:1428] Setting key 25e6cd1770988b6b776d39b36f0d6458
Sep 25 00:53:29 dist-test-slave-rpk1 krb5kdc[20197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:29.974889 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:29.961288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:42171 (local address 127.16.202.62:33529)
0925 00:53:29.961436 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:29.961440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:29.962373 (+ 933us) server_negotiation.cc:407] Connection header received
0925 00:53:29.963763 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:29.963766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:29.963826 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:29.963903 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:29.965617 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:29.966315 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:29.967616 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:29.967854 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:29.967955 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:29.970724 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:29.970727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:29.970729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:29.970758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:29.972315 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:29.973049 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:29.973053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:29.973054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:29.973140 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:29.973555 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:29.973557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:29.973558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:29.973804 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:29.973891 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:29.974604 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 00:53:29.974759 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:53:29.975742 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:29.961748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33529 (local address 127.16.202.1:42171)
0925 00:53:29.962221 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:29.962266 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:29.963537 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:29.964113 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:29.964121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:29.964640 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:29.965442 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:29.965465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:29.966495 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:29.966497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:29.967484 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:29.967495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:29.968011 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:29.968015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:29.968209 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:29.968815 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:29.968834 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:29.970488 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:29.972453 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:29.972461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:29.972482 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:29.972925 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:29.973234 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:29.973238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:29.973240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:29.973446 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:29.973923 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:29.973931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:29.974307 (+ 376us) client_negotiation.cc:769] Sending connection context
0925 00:53:29.974542 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 00:53:29.974872 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:29.977154 20433 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33529
I20260925 00:53:29.977448 20433 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:29.977952 20433 heartbeater.cc:511] Master 127.16.202.62:33529 requested a full tablet report, sending...
I20260925 00:53:29.979653 20242 ts_manager.cc:194] Registered new tserver with Master: 6f6cdca219974e1abfb2bb16bc9ec43e (127.16.202.1:34519)
I20260925 00:53:29.980852 20242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:42171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:29.994123 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33529
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:30.108729 20438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:30.109026 20438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:30.109076 20438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:30.113261 20438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:30.113340 20438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:30.113418 20438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:30.118909 20438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:30.120115 20438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:30.121251 20438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:30.131778 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:30.131778 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:30.131868 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:30.132161 20438 server_base.cc:1034] running on GCE node
I20260925 00:53:30.132622 20438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:30.133397 20438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:30.134588 20438 hybrid_clock.cc:648] HybridClock initialized: now 1790297610134571 us; error 29 us; skew 500 ppm
Sep 25 00:53:30 dist-test-slave-rpk1 krb5kdc[20197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:30.137722 20438 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:30.138885 20438 webserver.cc:460] Webserver started at http://127.16.202.2:37505/ using document root <none> and password file <none>
I20260925 00:53:30.139494 20438 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:30.139564 20438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:30.139776 20438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:30.142096 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "e59467aa9a0b4ba78d19a8c99a98848c"
format_stamp: "Formatted at 2026-09-25 00:53:30 on dist-test-slave-rpk1"
server_key: "77535e641762d1861da3404d6f9ec076"
server_key_iv: "fbbfc3717a1ef898f451eb99b550005c"
server_key_version: "encryptionkey@0"
I20260925 00:53:30.142717 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "e59467aa9a0b4ba78d19a8c99a98848c"
format_stamp: "Formatted at 2026-09-25 00:53:30 on dist-test-slave-rpk1"
server_key: "77535e641762d1861da3404d6f9ec076"
server_key_iv: "fbbfc3717a1ef898f451eb99b550005c"
server_key_version: "encryptionkey@0"
I20260925 00:53:30.146956 20438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:30.149838 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:30.151199 20438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:30.151335 20438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "e59467aa9a0b4ba78d19a8c99a98848c"
format_stamp: "Formatted at 2026-09-25 00:53:30 on dist-test-slave-rpk1"
server_key: "77535e641762d1861da3404d6f9ec076"
server_key_iv: "fbbfc3717a1ef898f451eb99b550005c"
server_key_version: "encryptionkey@0"
I20260925 00:53:30.151455 20438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:30.173136 20438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:30.176335 20438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:30.176515 20438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:30.177192 20438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:30.178045 20438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:30.178090 20438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:30.178130 20438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:30.178157 20438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:30.187148 20438 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:34419
I20260925 00:53:30.187220 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:34419 every 8 connection(s)
I20260925 00:53:30.188124 20438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:30.191125 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20438
I20260925 00:53:30.191262 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:30.191591 17192 external_mini_cluster.cc:1428] Setting key 5d79744e3d48fbac37896a6745b4ea5c
Sep 25 00:53:30 dist-test-slave-rpk1 krb5kdc[20197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:30.203377 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:30.190140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:52567 (local address 127.16.202.62:33529)
0925 00:53:30.190325 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:30.190328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:30.191314 (+ 986us) server_negotiation.cc:407] Connection header received
0925 00:53:30.192990 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:30.192993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:30.193042 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:30.193128 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:30.194714 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.195540 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.196717 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.196934 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.197023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:30.199578 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:30.199581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:30.199583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:30.199608 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:30.201155 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:30.201706 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:30.201710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:30.201712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:30.201813 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:30.202208 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:30.202210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:30.202212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:30.202407 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:30.202485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:30.203053 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 00:53:30.203192 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:30.204299 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:30.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33529 (local address 127.16.202.2:52567)
0925 00:53:30.191079 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:30.191146 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:30.192779 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:30.193340 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:30.193350 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:30.193846 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:30.194581 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:30.194600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.195671 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.195673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:30.196566 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:30.196575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.197077 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.197082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:30.197301 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:30.197945 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:30.197962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:30.199408 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:30.201317 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:30.201323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:30.201337 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:30.201604 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:30.201953 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:30.201956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:30.201957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:30.202111 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:30.202516 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:30.202521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:30.202792 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 00:53:30.202966 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 00:53:30.203254 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20260925 00:53:30.205502 20567 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33529
I20260925 00:53:30.205798 20567 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:30.206429 20567 heartbeater.cc:511] Master 127.16.202.62:33529 requested a full tablet report, sending...
I20260925 00:53:30.207600 20242 ts_manager.cc:194] Registered new tserver with Master: e59467aa9a0b4ba78d19a8c99a98848c (127.16.202.2:34419)
I20260925 00:53:30.208192 20242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:52567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:30.220782 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33529
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:30.334887 20572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:30.335171 20572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:30.335214 20572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:30.339514 20572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:30.339589 20572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:30.339655 20572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:30.345314 20572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:30.346498 20572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:30.347570 20572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:30.357441 20578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:30.357460 20577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:30.357609 20572 server_base.cc:1034] running on GCE node
W20260925 00:53:30.357460 20580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:30.358266 20572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:30.358986 20572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:30.360180 20572 hybrid_clock.cc:648] HybridClock initialized: now 1790297610360138 us; error 55 us; skew 500 ppm
Sep 25 00:53:30 dist-test-slave-rpk1 krb5kdc[20197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:30.363111 20572 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:30.364084 20572 webserver.cc:460] Webserver started at http://127.16.202.3:33727/ using document root <none> and password file <none>
I20260925 00:53:30.364655 20572 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:30.364763 20572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:30.364979 20572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:30.367230 20572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "71ed362d16c04b3885ddce1f4c655eae"
format_stamp: "Formatted at 2026-09-25 00:53:30 on dist-test-slave-rpk1"
server_key: "c9d547602362d58c0d547704b182bb2a"
server_key_iv: "32f866ec81c8a892fe006088c3294b19"
server_key_version: "encryptionkey@0"
I20260925 00:53:30.367775 20572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "71ed362d16c04b3885ddce1f4c655eae"
format_stamp: "Formatted at 2026-09-25 00:53:30 on dist-test-slave-rpk1"
server_key: "c9d547602362d58c0d547704b182bb2a"
server_key_iv: "32f866ec81c8a892fe006088c3294b19"
server_key_version: "encryptionkey@0"
I20260925 00:53:30.371884 20572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 00:53:30.374961 20587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:30.376379 20572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 00:53:30.376493 20572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "71ed362d16c04b3885ddce1f4c655eae"
format_stamp: "Formatted at 2026-09-25 00:53:30 on dist-test-slave-rpk1"
server_key: "c9d547602362d58c0d547704b182bb2a"
server_key_iv: "32f866ec81c8a892fe006088c3294b19"
server_key_version: "encryptionkey@0"
I20260925 00:53:30.376622 20572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:30.388617 20572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:30.392266 20572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:30.392454 20572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:30.393355 20572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:30.394259 20572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:30.394308 20572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:30.394347 20572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:30.394361 20572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:30.402736 20572 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:33627
I20260925 00:53:30.402805 20700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:33627 every 8 connection(s)
I20260925 00:53:30.403650 20572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:30.407538 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20572
I20260925 00:53:30.407687 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:30.408061 17192 external_mini_cluster.cc:1428] Setting key e3ff6d4a0948ffa6277e5d2e9ba89100
Sep 25 00:53:30 dist-test-slave-rpk1 krb5kdc[20197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:30.419322 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:30.405599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:35417 (local address 127.16.202.62:33529)
0925 00:53:30.405766 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:30.405771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:30.406725 (+ 954us) server_negotiation.cc:407] Connection header received
0925 00:53:30.408558 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:30.408562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:30.408626 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:30.408751 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:30.410152 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.410751 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.411981 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.412243 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.412350 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:30.414995 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:30.414999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:30.415001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:30.415028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:30.416436 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:30.417684 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:30.417688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:30.417690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:30.417779 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:30.418152 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:30.418156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:30.418157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:30.418376 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:30.418467 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:30.419021 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 00:53:30.419154 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:53:30.420239 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:30.406046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33529 (local address 127.16.202.3:35417)
0925 00:53:30.406564 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:30.406610 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:30.408304 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:30.408937 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:30.408945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:30.409444 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:30.409998 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:30.410014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.410888 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.410891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:30.411847 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:30.411857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.412401 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.412406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:30.412597 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:30.413235 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:30.413253 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:30.414801 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:30.417223 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:30.417229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:30.417244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:30.417573 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:30.417891 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:30.417894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:30.417896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:30.418060 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:30.418479 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:30.418486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:30.418775 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 00:53:30.418963 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 00:53:30.419350 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260925 00:53:30.421512 20701 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33529
I20260925 00:53:30.421772 20701 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:30.422320 20701 heartbeater.cc:511] Master 127.16.202.62:33529 requested a full tablet report, sending...
I20260925 00:53:30.423540 20241 ts_manager.cc:194] Registered new tserver with Master: 71ed362d16c04b3885ddce1f4c655eae (127.16.202.3:33627)
I20260925 00:53:30.424175 20241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:35417
I20260925 00:53:30.434314 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:53:30 dist-test-slave-rpk1 krb5kdc[20197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297610, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 00:53:30 dist-test-slave-rpk1 krb5kdc[20197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297610, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:30.512532 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:30.503730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.16.202.62:33529)
0925 00:53:30.503895 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:30.503901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:30.504088 (+ 187us) server_negotiation.cc:407] Connection header received
0925 00:53:30.504343 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:30.504347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:30.504401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:30.504484 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:30.505466 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.506038 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.507018 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:30.507242 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:30.507341 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:30.509778 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:30.509781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:30.509783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:30.509814 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:30.511044 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:30.511456 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:30.511459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:30.511460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:30.511542 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:30.511754 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:30.511756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:30.511757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:30.512001 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:30.512103 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:30.512267 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 00:53:30.512373 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 00:53:30.512969 20241 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47722
I20260925 00:53:30.513761 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20304
I20260925 00:53:30.518770 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20438
I20260925 00:53:30.523964 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20572
I20260925 00:53:30.528668 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20210
2026-09-25T00:53:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2142 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:32 dist-test-slave-rpk1 krb5kdc[20713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:32 dist-test-slave-rpk1 krb5kdc[20713](info): set up 2 sockets
Sep 25 00:53:32 dist-test-slave-rpk1 krb5kdc[20713](info): commencing operation
krb5kdc: starting...
W20260925 00:53:33.471463 17192 mini_kdc.cc:121] Time spent starting KDC: real 2.922s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 00:53:33.649158 17192 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.178s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 00:53:33.849229 17192 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.200s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 00:53:34.021421 17192 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.172s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/test-user.keytab.
W20260925 00:53:34.267570 17192 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.246s user 0.002s sys 0.000s
Sep 25 00:53:34 dist-test-slave-rpk1 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:34.312054 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40571
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:34.424058 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:34.424363 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:34.424422 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:34.428486 20726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:34.428556 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:34.428583 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:34.428606 20726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:34.428625 20726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:34.434072 20726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:40571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:34.435214 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:34.436286 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:34.444983 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:34.444971 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:34.444971 20731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:34.445297 20726 server_base.cc:1034] running on GCE node
I20260925 00:53:34.445842 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:34.446993 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:34.448177 20726 hybrid_clock.cc:648] HybridClock initialized: now 1790297614448127 us; error 62 us; skew 500 ppm
Sep 25 00:53:34 dist-test-slave-rpk1 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:34.455543 20726 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:34.456516 20726 webserver.cc:460] Webserver started at http://127.16.202.62:38241/ using document root <none> and password file <none>
I20260925 00:53:34.457113 20726 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:34.457181 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:34.457386 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:34.459455 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "d4f2460903de4d1dba22409b8e3246b7"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "b4602b1d7f2a3ae2824bcaeb3b54a71c"
server_key_iv: "03822fc6bf55d1d54e0702a838b734f4"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.459986 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "d4f2460903de4d1dba22409b8e3246b7"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "b4602b1d7f2a3ae2824bcaeb3b54a71c"
server_key_iv: "03822fc6bf55d1d54e0702a838b734f4"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.463711 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:53:34.466236 20741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:34.467522 20726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:34.467649 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "d4f2460903de4d1dba22409b8e3246b7"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "b4602b1d7f2a3ae2824bcaeb3b54a71c"
server_key_iv: "03822fc6bf55d1d54e0702a838b734f4"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.467747 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:34.505263 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:34.508524 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:34.508744 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:34.515539 20726 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:40571
I20260925 00:53:34.515581 20793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:40571 every 8 connection(s)
I20260925 00:53:34.516536 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:34.518651 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20726
I20260925 00:53:34.518774 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:34.519084 17192 external_mini_cluster.cc:1428] Setting key 9e4a0137550010c8a861e0c1117e8d36
I20260925 00:53:34.519939 20794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:34.526911 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7: Bootstrap starting.
Sep 25 00:53:34 dist-test-slave-rpk1 krb5kdc[20713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:34.529633 20794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:34.530400 20794 log.cc:824] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:34.532837 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7: No bootstrap required, opened a new log
I20260925 00:53:34.533946 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:34.520507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.16.202.62:40571)
0925 00:53:34.520947 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:34.520966 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:34.521008 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:53:34.521828 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:34.521851 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:34.522270 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:34.522641 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:34.523274 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:34.524842 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:34.525682 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:34.526364 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:34.526511 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:34.529167 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:34.529173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:34.529187 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:34.529214 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:34.530909 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:34.531352 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:34.531358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:34.531364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:34.531504 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:34.531724 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:34.531727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:34.531728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:34.532224 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:34.532413 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:34.532652 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 00:53:34.532975 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":155,"threads_started":1}
I20260925 00:53:34.537056 20794 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f2460903de4d1dba22409b8e3246b7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40571 } }
I20260925 00:53:34.537297 20794 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:34.537341 20794 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f2460903de4d1dba22409b8e3246b7, State: Initialized, Role: FOLLOWER
I20260925 00:53:34.537706 20794 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f2460903de4d1dba22409b8e3246b7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40571 } }
I20260925 00:53:34.537828 20794 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:34.537878 20794 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:34.537950 20794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:34.538870 20794 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f2460903de4d1dba22409b8e3246b7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40571 } }
I20260925 00:53:34.539194 20794 leader_election.cc:302] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f2460903de4d1dba22409b8e3246b7; no voters:
I20260925 00:53:34.539491 20794 leader_election.cc:288] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:34.539677 20799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:34.539911 20799 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [term 1 LEADER]: Becoming Leader. State: Replica: d4f2460903de4d1dba22409b8e3246b7, State: Running, Role: LEADER
I20260925 00:53:34.540264 20799 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f2460903de4d1dba22409b8e3246b7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40571 } }
I20260925 00:53:34.540961 20794 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:34.542044 20800 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f2460903de4d1dba22409b8e3246b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f2460903de4d1dba22409b8e3246b7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40571 } } }
I20260925 00:53:34.542199 20800 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:34.542482 20801 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f2460903de4d1dba22409b8e3246b7. Latest consensus state: current_term: 1 leader_uuid: "d4f2460903de4d1dba22409b8e3246b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f2460903de4d1dba22409b8e3246b7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40571 } } }
I20260925 00:53:34.542577 20801 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:34.542694 20808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:34.546111 20808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:34.552754 20808 catalog_manager.cc:1347] Generated new cluster ID: fc623511e12c41159535c792b832dcce
I20260925 00:53:34.552855 20808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:34.574723 20808 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:34.575877 20808 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:34.584028 20808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f2460903de4d1dba22409b8e3246b7: Generated new TSK 0
I20260925 00:53:34.584888 20808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:34.617199 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40571
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:34.733927 20820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:34.734225 20820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:34.734298 20820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:34.738837 20820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:34.738933 20820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:34.739024 20820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:34.745064 20820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:34.746209 20820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:34.747285 20820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:34.757259 20825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:34.757263 20828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:34.757422 20820 server_base.cc:1034] running on GCE node
W20260925 00:53:34.757261 20826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:34.758093 20820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:34.758831 20820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:34.760013 20820 hybrid_clock.cc:648] HybridClock initialized: now 1790297614759986 us; error 42 us; skew 500 ppm
Sep 25 00:53:34 dist-test-slave-rpk1 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:34.763067 20820 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:34.764128 20820 webserver.cc:460] Webserver started at http://127.16.202.1:42751/ using document root <none> and password file <none>
I20260925 00:53:34.764760 20820 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:34.764843 20820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:34.765087 20820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:34.767264 20820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "b1fd9dd8ed6945f488eee12de5c43ec7"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "8321b6aeb231bf66e540e0648816c8e3"
server_key_iv: "f4464926e86f051e2cfffa4d5474086c"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.767869 20820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "b1fd9dd8ed6945f488eee12de5c43ec7"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "8321b6aeb231bf66e540e0648816c8e3"
server_key_iv: "f4464926e86f051e2cfffa4d5474086c"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.772099 20820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:34.774879 20835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:34.776224 20820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:34.776348 20820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "b1fd9dd8ed6945f488eee12de5c43ec7"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "8321b6aeb231bf66e540e0648816c8e3"
server_key_iv: "f4464926e86f051e2cfffa4d5474086c"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.776471 20820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:34.787097 20820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:34.790345 20820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:34.790546 20820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:34.791155 20820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:34.792028 20820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:34.792078 20820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:34.792132 20820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:34.792158 20820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:34.800534 20820 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:33655
I20260925 00:53:34.800561 20948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:33655 every 8 connection(s)
I20260925 00:53:34.801542 20820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:34.804086 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20820
I20260925 00:53:34.804200 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:34.804504 17192 external_mini_cluster.cc:1428] Setting key a90b9c84981b954ccf6aca4ea23ce2c9
Sep 25 00:53:34 dist-test-slave-rpk1 krb5kdc[20713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:34.815143 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:34.803374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:35785 (local address 127.16.202.62:40571)
0925 00:53:34.803519 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:34.803523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:34.804421 (+ 898us) server_negotiation.cc:407] Connection header received
0925 00:53:34.805972 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:34.805975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:34.806025 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:34.806096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:34.807318 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:34.807846 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:34.809005 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:34.809231 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:34.809320 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:34.811670 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:34.811674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:34.811676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:34.811700 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:34.813056 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:34.813587 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:34.813590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:34.813592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:34.813680 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:34.814044 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:34.814046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:34.814048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:34.814240 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:34.814332 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:34.814843 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 00:53:34.814976 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:34.816179 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:34.803757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40571 (local address 127.16.202.1:35785)
0925 00:53:34.804253 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:34.804308 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:34.805761 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:34.806287 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:34.806296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:34.806662 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:34.807178 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:34.807192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:34.807965 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:34.807968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:34.808898 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:34.808907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:34.809369 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:34.809373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:34.809534 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:34.810081 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:34.810098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:34.811504 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:34.813179 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:34.813186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:34.813205 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:34.813474 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:34.813771 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:34.813774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:34.813775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:34.813959 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:34.814322 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:34.814328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:34.814627 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 00:53:34.814791 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 00:53:34.815119 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260925 00:53:34.817659 20949 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40571
I20260925 00:53:34.817955 20949 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:34.818599 20949 heartbeater.cc:511] Master 127.16.202.62:40571 requested a full tablet report, sending...
I20260925 00:53:34.820284 20757 ts_manager.cc:194] Registered new tserver with Master: b1fd9dd8ed6945f488eee12de5c43ec7 (127.16.202.1:33655)
I20260925 00:53:34.821519 20757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:35785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:34.832170 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40571
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:34.945155 20954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:34.945458 20954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:34.945502 20954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:34.949651 20954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:34.949719 20954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:34.949782 20954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:34.955166 20954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:34.956230 20954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:34.957312 20954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:34.966991 20960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:34.966931 20959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:34.967236 20954 server_base.cc:1034] running on GCE node
W20260925 00:53:34.966931 20962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:34.967752 20954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:34.968456 20954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:34.969625 20954 hybrid_clock.cc:648] HybridClock initialized: now 1790297614969549 us; error 110 us; skew 500 ppm
Sep 25 00:53:34 dist-test-slave-rpk1 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:34.972452 20954 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:34.973415 20954 webserver.cc:460] Webserver started at http://127.16.202.2:44059/ using document root <none> and password file <none>
I20260925 00:53:34.974007 20954 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:34.974069 20954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:34.974278 20954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:34.976382 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "a89ff694bf344125884b7a18f7c8636e"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "6f629aa6d3ac2e58436a9986ecadefe3"
server_key_iv: "b6e8aca33325053cb570b54deebe5b16"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.976929 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "a89ff694bf344125884b7a18f7c8636e"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "6f629aa6d3ac2e58436a9986ecadefe3"
server_key_iv: "b6e8aca33325053cb570b54deebe5b16"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.980787 20954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 00:53:34.983218 20969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:34.984457 20954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:34.984565 20954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "a89ff694bf344125884b7a18f7c8636e"
format_stamp: "Formatted at 2026-09-25 00:53:34 on dist-test-slave-rpk1"
server_key: "6f629aa6d3ac2e58436a9986ecadefe3"
server_key_iv: "b6e8aca33325053cb570b54deebe5b16"
server_key_version: "encryptionkey@0"
I20260925 00:53:34.984699 20954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:34.995090 20954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:34.998121 20954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:34.998306 20954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:34.998888 20954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:34.999750 20954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:34.999797 20954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:34.999840 20954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:34.999866 20954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:35.008370 20954 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:42755
I20260925 00:53:35.008402 21082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:42755 every 8 connection(s)
I20260925 00:53:35.009306 20954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:35.018605 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 20954
I20260925 00:53:35.018711 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[20713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297614, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:35.018996 17192 external_mini_cluster.cc:1428] Setting key 4548b08cf98604726940b3acc687c5c9
I20260925 00:53:35.023804 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:35.011115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:55143 (local address 127.16.202.62:40571)
0925 00:53:35.011233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:35.011237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:35.012152 (+ 915us) server_negotiation.cc:407] Connection header received
0925 00:53:35.013476 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:35.013479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:35.013528 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:35.013601 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:35.014767 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.015350 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.016560 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.016829 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.016921 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:35.019688 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:35.019691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:35.019693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:35.019720 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:35.021413 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.022056 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.022059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.022060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.022150 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.022607 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.022610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.022612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.022865 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:35.022967 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:35.023545 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 00:53:35.023671 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:53:35.025038 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:35.011513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40571 (local address 127.16.202.2:55143)
0925 00:53:35.011996 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:35.012039 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:35.013276 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:35.013771 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:35.013778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:35.014132 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:35.014617 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:35.014632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.015484 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.015487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:35.016432 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:35.016442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.017011 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.017014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:35.017182 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:35.017810 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:35.017827 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:35.019493 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:35.021541 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:35.021549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:35.021570 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:35.021933 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:35.022349 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:35.022352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:35.022354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:35.022520 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:35.022970 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:35.022976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:35.023260 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 00:53:35.023485 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 00:53:35.024001 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260925 00:53:35.026417 21083 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40571
I20260925 00:53:35.026682 21083 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:35.027235 21083 heartbeater.cc:511] Master 127.16.202.62:40571 requested a full tablet report, sending...
I20260925 00:53:35.028940 20757 ts_manager.cc:194] Registered new tserver with Master: a89ff694bf344125884b7a18f7c8636e (127.16.202.2:42755)
I20260925 00:53:35.029753 20757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:55143
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:35.053064 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40571
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:35.169790 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:35.170078 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:35.170122 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:35.174296 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:35.174360 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:35.174417 21088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:35.179673 21088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:35.180773 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:35.181833 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:35.191121 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:35.191207 21088 server_base.cc:1034] running on GCE node
W20260925 00:53:35.191161 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:35.191124 21096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:35.191828 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:35.192476 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:35.193639 21088 hybrid_clock.cc:648] HybridClock initialized: now 1790297615193613 us; error 42 us; skew 500 ppm
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297615, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:35.196225 21088 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:35.197166 21088 webserver.cc:460] Webserver started at http://127.16.202.3:37663/ using document root <none> and password file <none>
I20260925 00:53:35.197706 21088 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:35.197769 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:35.197956 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:35.200048 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "9fb30703e4bd4d809b1c8f98e301ab52"
format_stamp: "Formatted at 2026-09-25 00:53:35 on dist-test-slave-rpk1"
server_key: "fcbf6196ac6c380a9e11ac1b8f22a53d"
server_key_iv: "d7d21543a12a3988bbe0e437be39abca"
server_key_version: "encryptionkey@0"
I20260925 00:53:35.200598 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "9fb30703e4bd4d809b1c8f98e301ab52"
format_stamp: "Formatted at 2026-09-25 00:53:35 on dist-test-slave-rpk1"
server_key: "fcbf6196ac6c380a9e11ac1b8f22a53d"
server_key_iv: "d7d21543a12a3988bbe0e437be39abca"
server_key_version: "encryptionkey@0"
I20260925 00:53:35.204337 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:35.207113 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:35.208431 21088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:35.208554 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "9fb30703e4bd4d809b1c8f98e301ab52"
format_stamp: "Formatted at 2026-09-25 00:53:35 on dist-test-slave-rpk1"
server_key: "fcbf6196ac6c380a9e11ac1b8f22a53d"
server_key_iv: "d7d21543a12a3988bbe0e437be39abca"
server_key_version: "encryptionkey@0"
I20260925 00:53:35.208700 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:35.229233 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:35.232522 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:35.232754 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:35.233387 21088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:35.234277 21088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:35.234324 21088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:35.234380 21088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:35.234411 21088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:35.243691 21088 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:41051
I20260925 00:53:35.243713 21216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:41051 every 8 connection(s)
I20260925 00:53:35.244589 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:35.250444 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21088
I20260925 00:53:35.250581 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:35.250833 17192 external_mini_cluster.cc:1428] Setting key d6954bbc86461220b43b8631a5088f17
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[20713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297615, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:35.258708 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:35.246561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:37313 (local address 127.16.202.62:40571)
0925 00:53:35.246753 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:35.246757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:35.247667 (+ 910us) server_negotiation.cc:407] Connection header received
0925 00:53:35.249077 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:35.249081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:35.249141 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:35.249217 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:35.250503 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.251103 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.252610 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.252966 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.253072 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:35.255523 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:35.255526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:35.255528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:35.255552 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:35.256926 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.257388 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.257391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.257392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.257474 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.257764 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.257766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.257768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.257944 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:35.258017 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:35.258479 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 00:53:35.258586 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 00:53:35.259537 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:35.246964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40571 (local address 127.16.202.3:37313)
0925 00:53:35.247495 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:35.247548 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:35.248855 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:35.249422 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:35.249430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:35.249809 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:35.250353 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:35.250370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.251241 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.251245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:35.252483 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:35.252493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.253114 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.253118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:35.253296 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:35.253909 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:35.253925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:35.255350 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:35.257025 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:35.257029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:35.257043 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:35.257286 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:35.257551 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:35.257553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:35.257555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:35.257686 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:35.258020 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:35.258025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:35.258260 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 00:53:35.258422 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 00:53:35.258677 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":120,"threads_started":1}
I20260925 00:53:35.260751 21217 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40571
I20260925 00:53:35.260972 21217 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:35.261478 21217 heartbeater.cc:511] Master 127.16.202.62:40571 requested a full tablet report, sending...
I20260925 00:53:35.262493 20757 ts_manager.cc:194] Registered new tserver with Master: 9fb30703e4bd4d809b1c8f98e301ab52 (127.16.202.3:41051)
I20260925 00:53:35.263036 20757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:37313
I20260925 00:53:35.264722 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[20713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:35.291849 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:35.281016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.16.202.1:33655)
0925 00:53:35.281316 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:35.281331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:35.281357 (+ 26us) server_negotiation.cc:407] Connection header received
0925 00:53:35.281415 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:35.281420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:35.281614 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:35.281733 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:35.282689 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.283537 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.284456 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.284928 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.285041 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:35.287432 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:35.287443 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:35.287458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:35.287489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:35.289585 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.290093 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.290100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.290101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.290198 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.290580 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.290583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.290584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.290994 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:35.291218 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:35.291462 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 00:53:35.291638 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":123,"threads_started":1}
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[20713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:35.304098 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:35.295192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.16.202.62:40571)
0925 00:53:35.295385 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:35.295389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:35.295406 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:53:35.295505 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:35.295507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:35.295552 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:35.295627 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:35.296414 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.297067 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.297960 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:35.298193 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:35.298290 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:35.301035 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:35.301038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:35.301041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:35.301072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:35.302518 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.302956 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.302959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.302960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.303041 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:35.303304 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:35.303306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:35.303307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:35.303567 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:35.303669 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:35.303852 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 00:53:35.303948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260925 00:53:35.304553 20757 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52070
I20260925 00:53:35.305224 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20820
I20260925 00:53:35.310009 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20954
I20260925 00:53:35.314404 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21088
I20260925 00:53:35.318924 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 20726
2026-09-25T00:53:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4790 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[21229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[21229](info): set up 2 sockets
Sep 25 00:53:35 dist-test-slave-rpk1 krb5kdc[21229](info): commencing operation
krb5kdc: starting...
W20260925 00:53:36.284927 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:36 dist-test-slave-rpk1 krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:36.407682 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:44891
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:36.518182 21242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:36.518476 21242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:36.518519 21242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:36.522514 21242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:36.522574 21242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:36.522593 21242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:36.522609 21242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:36.522624 21242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:36.527962 21242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:36.529138 21242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:36.530184 21242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:36.538249 21250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:36.538298 21247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:36.538370 21242 server_base.cc:1034] running on GCE node
W20260925 00:53:36.538241 21248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:36.539081 21242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:36.540293 21242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:36.541481 21242 hybrid_clock.cc:648] HybridClock initialized: now 1790297616541464 us; error 27 us; skew 500 ppm
Sep 25 00:53:36 dist-test-slave-rpk1 krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297616, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:36.544492 21242 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:36.545347 21242 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:36.545425 21242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:36.545619 21242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:36.547732 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "33c1304ea1894e5c87c476784ba71e94"
format_stamp: "Formatted at 2026-09-25 00:53:36 on dist-test-slave-rpk1"
server_key: "41c352e60e7ac400c35e8168fdf28cf2"
server_key_iv: "5d3a1c15f0d62b8212136097df30616d"
server_key_version: "encryptionkey@0"
I20260925 00:53:36.548302 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "33c1304ea1894e5c87c476784ba71e94"
format_stamp: "Formatted at 2026-09-25 00:53:36 on dist-test-slave-rpk1"
server_key: "41c352e60e7ac400c35e8168fdf28cf2"
server_key_iv: "5d3a1c15f0d62b8212136097df30616d"
server_key_version: "encryptionkey@0"
I20260925 00:53:36.552140 21242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:36.554742 21256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:36.556018 21242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:36.556138 21242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "33c1304ea1894e5c87c476784ba71e94"
format_stamp: "Formatted at 2026-09-25 00:53:36 on dist-test-slave-rpk1"
server_key: "41c352e60e7ac400c35e8168fdf28cf2"
server_key_iv: "5d3a1c15f0d62b8212136097df30616d"
server_key_version: "encryptionkey@0"
I20260925 00:53:36.556243 21242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:36.567636 21242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:36.570837 21242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:36.571017 21242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:36.577791 21242 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:44891
I20260925 00:53:36.578070 21308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:44891 every 8 connection(s)
I20260925 00:53:36.578438 21242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:36.581485 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:36.583724 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21242
I20260925 00:53:36.583858 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:36.584152 17192 external_mini_cluster.cc:1428] Setting key 6be978cc2450ee2ae974ab42d7d8a6d8
I20260925 00:53:36.588116 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94: Bootstrap starting.
I20260925 00:53:36.590945 21309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:36.591806 21309 log.cc:824] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:36.594067 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94: No bootstrap required, opened a new log
Sep 25 00:53:36 dist-test-slave-rpk1 krb5kdc[21229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:36.598209 21309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1304ea1894e5c87c476784ba71e94" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44891 } }
I20260925 00:53:36.598464 21309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:36.598521 21309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c1304ea1894e5c87c476784ba71e94, State: Initialized, Role: FOLLOWER
I20260925 00:53:36.598968 21309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1304ea1894e5c87c476784ba71e94" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44891 } }
I20260925 00:53:36.599110 21309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:36.599174 21309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:36.599274 21309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:36.599285 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:36.586066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.16.202.62:44891)
0925 00:53:36.586558 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:36.586580 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:36.586629 (+ 49us) server_negotiation.cc:407] Connection header received
0925 00:53:36.587348 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:36.587365 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:36.587713 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:36.588015 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:36.588934 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:36.590399 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:36.591543 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:36.592081 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:36.592250 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:36.594850 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:36.594855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:36.594869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:36.594896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:36.596417 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:36.596904 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:36.596909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:36.596913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:36.597018 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:36.597295 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:36.597297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:36.597298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:36.597675 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:36.597825 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:36.598088 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 00:53:36.598416 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260925 00:53:36.600395 21309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1304ea1894e5c87c476784ba71e94" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44891 } }
I20260925 00:53:36.600757 21309 leader_election.cc:302] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c1304ea1894e5c87c476784ba71e94; no voters:
I20260925 00:53:36.601047 21309 leader_election.cc:288] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:36.601145 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:36.601347 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [term 1 LEADER]: Becoming Leader. State: Replica: 33c1304ea1894e5c87c476784ba71e94, State: Running, Role: LEADER
I20260925 00:53:36.601716 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1304ea1894e5c87c476784ba71e94" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44891 } }
I20260925 00:53:36.602260 21309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:36.603355 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c1304ea1894e5c87c476784ba71e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1304ea1894e5c87c476784ba71e94" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44891 } } }
I20260925 00:53:36.603369 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c1304ea1894e5c87c476784ba71e94. Latest consensus state: current_term: 1 leader_uuid: "33c1304ea1894e5c87c476784ba71e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1304ea1894e5c87c476784ba71e94" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44891 } } }
I20260925 00:53:36.603575 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:36.603595 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:36.603958 21324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:36.607009 21324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:36.612825 21324 catalog_manager.cc:1347] Generated new cluster ID: e7d88935b27c4c63906dab3d0215efeb
I20260925 00:53:36.612905 21324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:36.621996 21324 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:36.623013 21324 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:36.631048 21324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94: Generated new TSK 0
I20260925 00:53:36.631907 21324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:36.667101 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44891
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:36.783327 21335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:36.783609 21335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:36.783661 21335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:36.788014 21335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:36.788098 21335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:36.788163 21335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:36.793779 21335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:36.794929 21335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:36.795966 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:36.805442 21343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:36.805444 21341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:36.805444 21340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:36.805804 21335 server_base.cc:1034] running on GCE node
I20260925 00:53:36.806191 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:36.806804 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:36.807969 21335 hybrid_clock.cc:648] HybridClock initialized: now 1790297616807949 us; error 31 us; skew 500 ppm
Sep 25 00:53:36 dist-test-slave-rpk1 krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297616, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:36.810711 21335 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:36.811445 21335 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:36.811506 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:36.811671 21335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:36.813766 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "55a2d7eacc5042988aeb111252c9e437"
format_stamp: "Formatted at 2026-09-25 00:53:36 on dist-test-slave-rpk1"
server_key: "af703dcc1d3af4a79cff593a47d90ea6"
server_key_iv: "c8c9f176e3507046959eec7727a8e879"
server_key_version: "encryptionkey@0"
I20260925 00:53:36.814273 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "55a2d7eacc5042988aeb111252c9e437"
format_stamp: "Formatted at 2026-09-25 00:53:36 on dist-test-slave-rpk1"
server_key: "af703dcc1d3af4a79cff593a47d90ea6"
server_key_iv: "c8c9f176e3507046959eec7727a8e879"
server_key_version: "encryptionkey@0"
I20260925 00:53:36.818045 21335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:36.820700 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:36.822124 21335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:53:36.822289 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "55a2d7eacc5042988aeb111252c9e437"
format_stamp: "Formatted at 2026-09-25 00:53:36 on dist-test-slave-rpk1"
server_key: "af703dcc1d3af4a79cff593a47d90ea6"
server_key_iv: "c8c9f176e3507046959eec7727a8e879"
server_key_version: "encryptionkey@0"
I20260925 00:53:36.822408 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:36.840991 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:36.844348 21335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:36.844523 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:36.845168 21335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:36.846082 21335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:36.846134 21335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:36.846175 21335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:36.846202 21335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:36.854812 21335 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:46541
I20260925 00:53:36.854827 21462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:46541 every 8 connection(s)
I20260925 00:53:36.855420 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:36.864363 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21335
I20260925 00:53:36.864495 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:36.864831 17192 external_mini_cluster.cc:1428] Setting key 855a17e63710de8db6d573106df3248c
Sep 25 00:53:36 dist-test-slave-rpk1 krb5kdc[21229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297616, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:36.869459 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:36.857204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:56947 (local address 127.16.202.62:44891)
0925 00:53:36.857373 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:36.857377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:36.858219 (+ 842us) server_negotiation.cc:407] Connection header received
0925 00:53:36.859567 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:36.859570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:36.859653 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:36.859719 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:36.860992 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:36.861589 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:36.862720 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:36.862953 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:36.863042 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:36.865814 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:36.865817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:36.865819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:36.865844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:36.867348 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:36.867833 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:36.867836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:36.867838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:36.867934 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:36.868325 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:36.868328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:36.868329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:36.868513 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:36.868588 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:36.869229 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 00:53:36.869335 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:36.870685 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:36.857588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:44891 (local address 127.16.202.1:56947)
0925 00:53:36.858056 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:36.858101 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:36.859329 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:36.859918 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:36.859926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:36.860287 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:36.860846 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:36.860861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:36.861714 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:36.861717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:36.862596 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:36.862606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:36.863100 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:36.863105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:36.863269 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:36.863850 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:36.863867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:36.865643 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:36.867462 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:36.867467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:36.867481 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:36.867748 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:36.868022 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:36.868025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:36.868027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:36.868238 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:36.868599 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:36.868606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:36.868973 (+ 367us) client_negotiation.cc:769] Sending connection context
0925 00:53:36.869194 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 00:53:36.869605 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260925 00:53:36.871999 21463 heartbeater.cc:347] Connected to a master server at 127.16.202.62:44891
I20260925 00:53:36.872234 21463 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:36.872721 21463 heartbeater.cc:511] Master 127.16.202.62:44891 requested a full tablet report, sending...
I20260925 00:53:36.874219 21273 ts_manager.cc:194] Registered new tserver with Master: 55a2d7eacc5042988aeb111252c9e437 (127.16.202.1:46541)
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:36.875475 21273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:56947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:36.893200 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44891
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:37.006341 21468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:37.006661 21468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:37.006708 21468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:37.010898 21468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:37.010970 21468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:37.011035 21468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:37.016500 21468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:37.017658 21468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:37.018705 21468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:37.027938 21474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:37.027938 21473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:37.027978 21476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:37.028316 21468 server_base.cc:1034] running on GCE node
I20260925 00:53:37.028746 21468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:37.029399 21468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:37.030570 21468 hybrid_clock.cc:648] HybridClock initialized: now 1790297617030548 us; error 35 us; skew 500 ppm
Sep 25 00:53:37 dist-test-slave-rpk1 krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:37.033372 21468 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:37.034102 21468 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:37.034169 21468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:37.034358 21468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:37.036391 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "74674733088644c99bfc5c4ae843756b"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "f32fbd62001e97f17e0f3da095d3e411"
server_key_iv: "c5dbad682984f0ba6f9e6a85906881a6"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.036972 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "74674733088644c99bfc5c4ae843756b"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "f32fbd62001e97f17e0f3da095d3e411"
server_key_iv: "c5dbad682984f0ba6f9e6a85906881a6"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.040975 21468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:53:37.043467 21482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.044870 21468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:37.044999 21468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "74674733088644c99bfc5c4ae843756b"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "f32fbd62001e97f17e0f3da095d3e411"
server_key_iv: "c5dbad682984f0ba6f9e6a85906881a6"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.045106 21468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:37.060603 21468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:37.063865 21468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:37.064052 21468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:37.064651 21468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:37.065560 21468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:37.065616 21468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.065672 21468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:37.065699 21468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.074249 21468 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:37889
I20260925 00:53:37.074282 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:37889 every 8 connection(s)
I20260925 00:53:37.074904 21468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:37.079789 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21468
I20260925 00:53:37.079924 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:37.080228 17192 external_mini_cluster.cc:1428] Setting key d90597482a34bddb5425178abff9ce3b
Sep 25 00:53:37 dist-test-slave-rpk1 krb5kdc[21229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:37.089888 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.076827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:39989 (local address 127.16.202.62:44891)
0925 00:53:37.076970 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.076974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.078143 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 00:53:37.079067 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.079070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.079121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.079199 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:37.080787 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.081317 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.082839 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.083078 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.083163 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.085974 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:37.085979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:37.085982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:37.086016 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:37.087691 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:37.088234 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:37.088238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:37.088240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:37.088331 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:37.088661 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:37.088664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:37.088666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:37.088945 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:37.089047 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.089574 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.089709 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:37.090648 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.077183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:44891 (local address 127.16.202.2:39989)
0925 00:53:37.077995 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.078038 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.078860 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.079394 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.079402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.079886 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:37.080603 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.080625 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.081440 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.081444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.082716 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.082727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.083230 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.083234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.083449 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.084019 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:37.084037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:37.085754 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:37.087823 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:37.087829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:37.087843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:37.088120 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:37.088419 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:37.088422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:37.088423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:37.088566 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:37.089040 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:37.089046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:37.089312 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.089469 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.089750 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260925 00:53:37.091914 21596 heartbeater.cc:347] Connected to a master server at 127.16.202.62:44891
I20260925 00:53:37.092171 21596 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:37.092746 21596 heartbeater.cc:511] Master 127.16.202.62:44891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:37.093854 21273 ts_manager.cc:194] Registered new tserver with Master: 74674733088644c99bfc5c4ae843756b (127.16.202.2:37889)
I20260925 00:53:37.094587 21273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:39989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:37.108032 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44891
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:37.216941 21601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:37.217231 21601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:37.217303 21601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:37.221359 21601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:37.221421 21601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:37.221505 21601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:37.226994 21601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:37.228070 21601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:37.229120 21601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:37.238255 21606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:37.238389 21609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:37.238413 21601 server_base.cc:1034] running on GCE node
W20260925 00:53:37.238394 21607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:37.239001 21601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:37.239624 21601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:37.240751 21601 hybrid_clock.cc:648] HybridClock initialized: now 1790297617240726 us; error 48 us; skew 500 ppm
Sep 25 00:53:37 dist-test-slave-rpk1 krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:37.243574 21601 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:37.244295 21601 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:37.244364 21601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:37.244588 21601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:37.246699 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "c952c2f854fc41cdbececa924677b366"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "475db29a3d61e5a2d92a20f8917be3d8"
server_key_iv: "da45b484c6636f9cccf5770cd23a6f96"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.247252 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "c952c2f854fc41cdbececa924677b366"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "475db29a3d61e5a2d92a20f8917be3d8"
server_key_iv: "da45b484c6636f9cccf5770cd23a6f96"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.251128 21601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:37.253685 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.254988 21601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:37.255126 21601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "c952c2f854fc41cdbececa924677b366"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "475db29a3d61e5a2d92a20f8917be3d8"
server_key_iv: "da45b484c6636f9cccf5770cd23a6f96"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.255228 21601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:37.276465 21601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:37.279861 21601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:37.280062 21601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:37.280709 21601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:37.281620 21601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:37.281668 21601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.281723 21601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:37.281751 21601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.290350 21601 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:46267
I20260925 00:53:37.290460 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:46267 every 8 connection(s)
I20260925 00:53:37.290973 21601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:37.294440 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21601
I20260925 00:53:37.294586 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:37.294931 17192 external_mini_cluster.cc:1428] Setting key 6d7798b0174bcf88f3000ad2bb51c9f2
Sep 25 00:53:37 dist-test-slave-rpk1 krb5kdc[21229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:37.305572 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.292860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:45683 (local address 127.16.202.62:44891)
0925 00:53:37.293022 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.293025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.294445 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 00:53:37.295602 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.295606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.295667 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.295755 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:37.297108 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.297632 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.298952 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.299168 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.299244 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.301841 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:37.301844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:37.301846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:37.301873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:37.303223 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:37.303732 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:37.303736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:37.303738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:37.303817 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:37.304256 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:37.304258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:37.304260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:37.304503 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:37.304585 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.305276 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.305445 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:53:37.306337 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.293301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:44891 (local address 127.16.202.3:45683)
0925 00:53:37.294246 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.294312 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.295381 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.295970 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.295980 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.296398 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:37.296965 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.296980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.297767 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.297770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.298834 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.298844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.299331 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.299335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.299487 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.300044 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:37.300061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:37.301626 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:37.303334 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:37.303339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:37.303354 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:37.303617 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:37.303976 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:37.303979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:37.303981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:37.304135 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:37.304620 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:37.304627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:37.304961 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.305130 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.305424 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":121,"threads_started":1}
I20260925 00:53:37.307499 21729 heartbeater.cc:347] Connected to a master server at 127.16.202.62:44891
I20260925 00:53:37.307753 21729 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:37.308243 21729 heartbeater.cc:511] Master 127.16.202.62:44891 requested a full tablet report, sending...
I20260925 00:53:37.309257 21273 ts_manager.cc:194] Registered new tserver with Master: c952c2f854fc41cdbececa924677b366 (127.16.202.3:46267)
I20260925 00:53:37.309804 21273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:45683
I20260925 00:53:37.321046 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:53:37.330545 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.322991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.16.202.62:44891)
0925 00:53:37.323131 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.323136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.323308 (+ 172us) server_negotiation.cc:407] Connection header received
0925 00:53:37.323479 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.323482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.323528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.323599 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:37.324271 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.324882 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.325753 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.325967 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.326051 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.327313 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:37.327317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:37.327319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:37.327350 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:37.328703 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:37.329112 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:37.329115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:37.329116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:37.329204 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:37.329624 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:37.329627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:37.329628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:37.329900 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:37.330022 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.330254 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.330372 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:53:37.333775 21273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:37.336915 21273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:37.351104 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.345570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37889 (local address 127.0.0.1:45542)
0925 00:53:37.346158 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.346215 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.346359 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.347093 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.347100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.347124 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:37.347256 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.347264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.348526 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.348529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.349781 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.349791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.350556 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.350560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.350666 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.350701 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.350811 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.350944 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260925 00:53:37.351109 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.345858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.16.202.2:37889)
0925 00:53:37.346294 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.346306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.346330 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:53:37.346554 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.346560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.346777 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.346925 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:37.347389 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.348404 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.349901 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.350444 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.350542 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.350699 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.350849 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.350956 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":134,"threads_started":1}
I20260925 00:53:37.352054 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.346310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46541 (local address 127.0.0.1:43198)
0925 00:53:37.347169 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.347182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.347251 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.347831 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.347834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.347852 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:37.347940 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.347946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.349277 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.349280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.350569 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.350580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.351634 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.351637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.351737 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.351762 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.351838 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.351920 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":61,"thread_start_us":132,"threads_started":1}
I20260925 00:53:37.352083 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.346755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.16.202.1:46541)
0925 00:53:37.347219 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.347227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.347248 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:53:37.347372 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.347377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.347521 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.347653 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:37.348057 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.349161 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.350707 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.351524 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.351644 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.351770 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.351883 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.351961 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":102,"threads_started":1}
I20260925 00:53:37.352597 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.345617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:46267 (local address 127.0.0.1:49146)
0925 00:53:37.346849 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.346863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.347068 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.347607 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.347610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.347627 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:37.347728 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.347735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.349251 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.349255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.350570 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.350580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.352205 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.352208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.352302 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.352323 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.352409 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.352472 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":79,"thread_start_us":95,"threads_started":1}
I20260925 00:53:37.352723 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.346286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.16.202.3:46267)
0925 00:53:37.346716 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.346727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.347173 (+ 446us) server_negotiation.cc:407] Connection header received
0925 00:53:37.347257 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.347263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.347471 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.347620 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:37.347861 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.349101 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.350707 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.351524 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.351632 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.351770 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.352443 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.352548 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":128,"threads_started":1}
I20260925 00:53:37.354691 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 14dd6794bedc4cd38c342dfd47e28a28 (DEFAULT_TABLE table=test-table [id=19178c031d20401da1b944c8a2db271b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:37.355043 21397 tablet_service.cc:1467] Processing CreateTablet for tablet 14dd6794bedc4cd38c342dfd47e28a28 (DEFAULT_TABLE table=test-table [id=19178c031d20401da1b944c8a2db271b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:37.355700 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dd6794bedc4cd38c342dfd47e28a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:37.355875 21397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dd6794bedc4cd38c342dfd47e28a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:37.358520 21663 tablet_service.cc:1467] Processing CreateTablet for tablet 14dd6794bedc4cd38c342dfd47e28a28 (DEFAULT_TABLE table=test-table [id=19178c031d20401da1b944c8a2db271b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:37.359661 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dd6794bedc4cd38c342dfd47e28a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:37.361905 21746 tablet_bootstrap.cc:492] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Bootstrap starting.
I20260925 00:53:37.363626 21747 tablet_bootstrap.cc:492] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437: Bootstrap starting.
I20260925 00:53:37.364207 21746 tablet_bootstrap.cc:654] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:37.365430 21746 log.cc:824] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:37.365980 21747 tablet_bootstrap.cc:654] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:37.366533 21749 tablet_bootstrap.cc:492] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Bootstrap starting.
I20260925 00:53:37.367148 21747 log.cc:824] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:37.367594 21746 tablet_bootstrap.cc:492] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: No bootstrap required, opened a new log
I20260925 00:53:37.367825 21746 ts_tablet_manager.cc:1397] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:53:37.368592 21749 tablet_bootstrap.cc:654] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:37.369256 21747 tablet_bootstrap.cc:492] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437: No bootstrap required, opened a new log
I20260925 00:53:37.369438 21747 ts_tablet_manager.cc:1397] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:53:37.369572 21749 log.cc:824] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:37.371021 21749 tablet_bootstrap.cc:492] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: No bootstrap required, opened a new log
I20260925 00:53:37.371186 21749 ts_tablet_manager.cc:1397] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:53:37.371734 21746 raft_consensus.cc:348] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.372043 21746 raft_consensus.cc:374] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:37.372110 21746 raft_consensus.cc:729] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74674733088644c99bfc5c4ae843756b, State: Initialized, Role: FOLLOWER
I20260925 00:53:37.372578 21746 consensus_queue.cc:260] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.373227 21747 raft_consensus.cc:348] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.373416 21596 heartbeater.cc:503] Master 127.16.202.62:44891 was elected leader, sending a full tablet report...
I20260925 00:53:37.373521 21747 raft_consensus.cc:374] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:37.373616 21746 ts_tablet_manager.cc:1428] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:53:37.373620 21747 raft_consensus.cc:729] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a2d7eacc5042988aeb111252c9e437, State: Initialized, Role: FOLLOWER
I20260925 00:53:37.374096 21747 consensus_queue.cc:260] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.374648 21463 heartbeater.cc:503] Master 127.16.202.62:44891 was elected leader, sending a full tablet report...
I20260925 00:53:37.374586 21749 raft_consensus.cc:348] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.374837 21749 raft_consensus.cc:374] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:37.374886 21749 raft_consensus.cc:729] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c952c2f854fc41cdbececa924677b366, State: Initialized, Role: FOLLOWER
I20260925 00:53:37.375003 21747 ts_tablet_manager.cc:1428] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 00:53:37.375331 21749 consensus_queue.cc:260] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.376186 21729 heartbeater.cc:503] Master 127.16.202.62:44891 was elected leader, sending a full tablet report...
I20260925 00:53:37.376283 21749 ts_tablet_manager.cc:1428] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:53:37.502674 21753 raft_consensus.cc:482] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:37.502893 21753 raft_consensus.cc:504] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.504150 21753 leader_election.cc:288] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74674733088644c99bfc5c4ae843756b (127.16.202.2:37889), c952c2f854fc41cdbececa924677b366 (127.16.202.3:46267)
I20260925 00:53:37.509172 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.504563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:60017 (local address 127.16.202.3:46267)
0925 00:53:37.504713 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.504717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.505053 (+ 336us) server_negotiation.cc:407] Connection header received
0925 00:53:37.505129 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.505132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.505180 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.505260 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:37.505558 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.506348 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.508152 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.508562 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.508657 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.508723 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.508983 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.509061 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:53:37.509191 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.504471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:46267 (local address 127.16.202.1:60017)
0925 00:53:37.504967 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.504978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.505051 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.505304 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.505307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.505318 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:37.505458 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.505463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.506493 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.506497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.508023 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.508036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.508740 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.508744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.508829 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.508856 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.508948 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.509030 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":109,"spinlock_wait_cycles":1408,"thread_start_us":132,"threads_started":1}
I20260925 00:53:37.509734 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.504471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37889 (local address 127.16.202.1:47977)
0925 00:53:37.504874 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.504889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:37.505028 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:37.505284 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:37.505287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:37.505304 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:37.505457 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.505463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.506493 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.506497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.508027 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:37.508036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.509300 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.509304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:37.509399 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.509422 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:37.509521 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:53:37.509591 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":143,"spinlock_wait_cycles":6400,"thread_start_us":137,"threads_started":1}
I20260925 00:53:37.509761 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.504666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:47977 (local address 127.16.202.2:37889)
0925 00:53:37.504801 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.504805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.504969 (+ 164us) server_negotiation.cc:407] Connection header received
0925 00:53:37.505129 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.505132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.505180 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.505260 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:37.505558 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.506366 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.508780 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.509175 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.509282 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.509328 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.509568 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.509656 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:53:37.509946 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dd6794bedc4cd38c342dfd47e28a28" candidate_uuid: "55a2d7eacc5042988aeb111252c9e437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c952c2f854fc41cdbececa924677b366" is_pre_election: true
I20260925 00:53:37.510278 21683 raft_consensus.cc:2454] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a2d7eacc5042988aeb111252c9e437 in term 0.
I20260925 00:53:37.510449 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dd6794bedc4cd38c342dfd47e28a28" candidate_uuid: "55a2d7eacc5042988aeb111252c9e437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74674733088644c99bfc5c4ae843756b" is_pre_election: true
I20260925 00:53:37.510751 21550 raft_consensus.cc:2454] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a2d7eacc5042988aeb111252c9e437 in term 0.
I20260925 00:53:37.510982 21353 leader_election.cc:302] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a2d7eacc5042988aeb111252c9e437, c952c2f854fc41cdbececa924677b366; no voters:
I20260925 00:53:37.511305 21753 raft_consensus.cc:2781] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:37.511408 21753 raft_consensus.cc:482] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:37.511461 21753 raft_consensus.cc:3037] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:37.512498 21753 raft_consensus.cc:504] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.512887 21753 leader_election.cc:288] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [CANDIDATE]: Term 1 election: Requested vote from peers 74674733088644c99bfc5c4ae843756b (127.16.202.2:37889), c952c2f854fc41cdbececa924677b366 (127.16.202.3:46267)
I20260925 00:53:37.513273 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dd6794bedc4cd38c342dfd47e28a28" candidate_uuid: "55a2d7eacc5042988aeb111252c9e437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c952c2f854fc41cdbececa924677b366"
I20260925 00:53:37.513283 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dd6794bedc4cd38c342dfd47e28a28" candidate_uuid: "55a2d7eacc5042988aeb111252c9e437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74674733088644c99bfc5c4ae843756b"
I20260925 00:53:37.513401 21683 raft_consensus.cc:3037] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:37.513393 21550 raft_consensus.cc:3037] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:37.514427 21683 raft_consensus.cc:2454] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a2d7eacc5042988aeb111252c9e437 in term 1.
I20260925 00:53:37.514444 21550 raft_consensus.cc:2454] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a2d7eacc5042988aeb111252c9e437 in term 1.
I20260925 00:53:37.514729 21352 leader_election.cc:302] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a2d7eacc5042988aeb111252c9e437, 74674733088644c99bfc5c4ae843756b; no voters:
I20260925 00:53:37.514894 21753 raft_consensus.cc:2781] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:37.515142 21753 raft_consensus.cc:686] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [term 1 LEADER]: Becoming Leader. State: Replica: 55a2d7eacc5042988aeb111252c9e437, State: Running, Role: LEADER
I20260925 00:53:37.515506 21753 consensus_queue.cc:237] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } }
I20260925 00:53:37.518648 21273 catalog_manager.cc:5495] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a2d7eacc5042988aeb111252c9e437 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "55a2d7eacc5042988aeb111252c9e437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a2d7eacc5042988aeb111252c9e437" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46541 } health_report { overall_health: HEALTHY } } }
W20260925 00:53:37.541899 21730 tablet.cc:2369] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:37.553830 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.550030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.16.202.1:46541)
0925 00:53:37.550179 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.550183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.550224 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:53:37.550310 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.550312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.550366 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.550452 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:37.550760 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.551290 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.552195 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:37.552453 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:37.552534 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:37.552947 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:37.553452 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:37.553553 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.553631 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.553688 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:53:37.559875 21683 raft_consensus.cc:1261] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 1 FOLLOWER]: Refusing update from remote peer 55a2d7eacc5042988aeb111252c9e437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:37.559897 21550 raft_consensus.cc:1261] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 1 FOLLOWER]: Refusing update from remote peer 55a2d7eacc5042988aeb111252c9e437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:37.560606 21757 consensus_queue.cc:1035] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74674733088644c99bfc5c4ae843756b" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:37.560849 21753 consensus_queue.cc:1035] T 14dd6794bedc4cd38c342dfd47e28a28 P 55a2d7eacc5042988aeb111252c9e437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c952c2f854fc41cdbececa924677b366" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:37.568908 21761 mvcc.cc:205] Tried to move back new op lower bound from 7333059041517686784 to 7333059041346121728. Current Snapshot: MvccSnapshot[applied={T|T < 7333059041517686784}]
I20260925 00:53:37.569077 21762 mvcc.cc:205] Tried to move back new op lower bound from 7333059041517686784 to 7333059041346121728. Current Snapshot: MvccSnapshot[applied={T|T < 7333059041517686784}]
I20260925 00:53:37.577342 21273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:37.577507 21273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:37.579829 21273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c1304ea1894e5c87c476784ba71e94: Sending DeleteTablet for 3 replicas of tablet 14dd6794bedc4cd38c342dfd47e28a28
I20260925 00:53:37.580713 21397 tablet_service.cc:1514] Processing DeleteTablet for tablet 14dd6794bedc4cd38c342dfd47e28a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:37 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:43198
I20260925 00:53:37.580741 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 14dd6794bedc4cd38c342dfd47e28a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:37 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:45542
I20260925 00:53:37.580772 21663 tablet_service.cc:1514] Processing DeleteTablet for tablet 14dd6794bedc4cd38c342dfd47e28a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:37 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49146
I20260925 00:53:37.581845 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21335
I20260925 00:53:37.582759 21772 tablet_replica.cc:330] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: stopping tablet replica
I20260925 00:53:37.583094 21772 raft_consensus.cc:2229] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:37.583431 21772 raft_consensus.cc:2258] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:37.583806 21771 tablet_replica.cc:330] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: stopping tablet replica
I20260925 00:53:37.584107 21771 raft_consensus.cc:2229] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:37.584437 21771 raft_consensus.cc:2258] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:37.585059 21772 ts_tablet_manager.cc:1905] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:37.585835 21771 ts_tablet_manager.cc:1905] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:37.587922 21772 ts_tablet_manager.cc:1918] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:37.588058 21772 log.cc:1196] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/14dd6794bedc4cd38c342dfd47e28a28
I20260925 00:53:37.588402 21772 ts_tablet_manager.cc:1939] T 14dd6794bedc4cd38c342dfd47e28a28 P c952c2f854fc41cdbececa924677b366: Deleting consensus metadata
I20260925 00:53:37.588836 21771 ts_tablet_manager.cc:1918] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:37.588992 21771 log.cc:1196] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/14dd6794bedc4cd38c342dfd47e28a28
I20260925 00:53:37.589319 21771 ts_tablet_manager.cc:1939] T 14dd6794bedc4cd38c342dfd47e28a28 P 74674733088644c99bfc5c4ae843756b: Deleting consensus metadata
I20260925 00:53:37.589910 21260 catalog_manager.cc:4841] TS c952c2f854fc41cdbececa924677b366 (127.16.202.3:46267): tablet 14dd6794bedc4cd38c342dfd47e28a28 (table test-table [id=19178c031d20401da1b944c8a2db271b]) successfully deleted
I20260925 00:53:37.590425 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21468
W20260925 00:53:37.590970 21259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:46541 (error 108)
I20260925 00:53:37.593465 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.593209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46541 (local address 127.0.0.1:43204)
0925 00:53:37.593316 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.593383 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 00:53:37.593641 21259 catalog_manager.cc:4584] TS 55a2d7eacc5042988aeb111252c9e437 (127.16.202.1:46541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14dd6794bedc4cd38c342dfd47e28a28: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46541: connect: Connection refused (error 111)
I20260925 00:53:37.597990 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21601
I20260925 00:53:37.598785 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.598160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37889 (local address 127.0.0.1:45544)
0925 00:53:37.598643 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:37.598697 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.2:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260925 00:53:37.602913 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21242
2026-09-25T00:53:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2283 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T00:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:37Z Disabled control of system clock
I20260925 00:53:37.631572 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33019
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:33019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 00:53:37.746872 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:37.747160 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:37.747198 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:37.751353 21778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:37.751437 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:37.751452 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:37.751467 21778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:37.751482 21778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:37.756518 21778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:33019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:37.757697 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:37.759120 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:37.768098 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:37.768133 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:37.768217 21778 server_base.cc:1034] running on GCE node
W20260925 00:53:37.768103 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:37.769044 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:37.770222 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:37.771407 21778 hybrid_clock.cc:648] HybridClock initialized: now 1790297617771390 us; error 29 us; skew 500 ppm
I20260925 00:53:37.773343 21778 webserver.cc:460] Webserver started at http://127.16.202.62:43417/ using document root <none> and password file <none>
I20260925 00:53:37.773919 21778 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:37.773988 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:37.774195 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:37.776298 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "2f2c443923d0493894a2e2de656d4fb1"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "b87c0c20f1addfbc30c484fdf8ffd307"
server_key_iv: "56766f8ef6dfd968a2cdf21ddb58ffa2"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.776887 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "2f2c443923d0493894a2e2de656d4fb1"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "b87c0c20f1addfbc30c484fdf8ffd307"
server_key_iv: "56766f8ef6dfd968a2cdf21ddb58ffa2"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.780467 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:37.782871 21792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:37.784097 21778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:37.784209 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "2f2c443923d0493894a2e2de656d4fb1"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "b87c0c20f1addfbc30c484fdf8ffd307"
server_key_iv: "56766f8ef6dfd968a2cdf21ddb58ffa2"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.784304 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:37.788847 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:37.789419 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:37.789569 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:37.796072 21778 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:33019
I20260925 00:53:37.796123 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:33019 every 8 connection(s)
I20260925 00:53:37.797065 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:37.797435 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21778
I20260925 00:53:37.797541 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:37.797798 17192 external_mini_cluster.cc:1428] Setting key 9256260adb87f5961aeeaed7d2d5f92d
I20260925 00:53:37.800050 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:37.803956 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:37.799240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.16.202.62:33019)
0925 00:53:37.800306 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:37.800323 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:37.800361 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:53:37.801066 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:37.801086 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:37.801456 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:37.801755 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:37.802623 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:37.802627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:37.802642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:37.802652 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:37.802708 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:37.802724 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:37.802868 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:37.802950 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 00:53:37.803200 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":518,"threads_started":1}
I20260925 00:53:37.806188 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1: Bootstrap starting.
I20260925 00:53:37.808332 21845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:37.809110 21845 log.cc:824] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:37.811117 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1: No bootstrap required, opened a new log
I20260925 00:53:37.814670 21845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c443923d0493894a2e2de656d4fb1" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33019 } }
I20260925 00:53:37.814913 21845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:37.814949 21845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2c443923d0493894a2e2de656d4fb1, State: Initialized, Role: FOLLOWER
I20260925 00:53:37.815392 21845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c443923d0493894a2e2de656d4fb1" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33019 } }
I20260925 00:53:37.815514 21845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:37.815578 21845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:37.815671 21845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:37.816601 21845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c443923d0493894a2e2de656d4fb1" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33019 } }
I20260925 00:53:37.816998 21845 leader_election.cc:302] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f2c443923d0493894a2e2de656d4fb1; no voters:
I20260925 00:53:37.817318 21845 leader_election.cc:288] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:37.817402 21850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:37.817587 21850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f2c443923d0493894a2e2de656d4fb1, State: Running, Role: LEADER
I20260925 00:53:37.817934 21850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c443923d0493894a2e2de656d4fb1" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33019 } }
I20260925 00:53:37.818688 21845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:37.819741 21851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f2c443923d0493894a2e2de656d4fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c443923d0493894a2e2de656d4fb1" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33019 } } }
I20260925 00:53:37.819872 21851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:37.820187 21852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f2c443923d0493894a2e2de656d4fb1. Latest consensus state: current_term: 1 leader_uuid: "2f2c443923d0493894a2e2de656d4fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c443923d0493894a2e2de656d4fb1" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33019 } } }
I20260925 00:53:37.820338 21852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:37.820255 21859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:37.823460 21859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:37.828974 21859 catalog_manager.cc:1347] Generated new cluster ID: b9ab8c31d320448896baa4ce6b3a6c03
I20260925 00:53:37.829061 21859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:37.838814 21859 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:37.839182 21859 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:37.847756 21859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1: Generated new TSK 0
I20260925 00:53:37.848551 21859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:53:37.853626 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33019
--builtin_ntp_servers=127.16.202.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 00:53:37.964290 21869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:37.964555 21869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:37.964610 21869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:37.968597 21869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:37.968657 21869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:37.968758 21869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:37.973587 21869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:37.974710 21869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:37.976135 21869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:37.985909 21869 server_base.cc:1034] running on GCE node
W20260925 00:53:37.985847 21877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:37.985862 21874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:37.985858 21875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:37.986459 21869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:37.987083 21869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:37.988245 21869 hybrid_clock.cc:648] HybridClock initialized: now 1790297617988199 us; error 55 us; skew 500 ppm
I20260925 00:53:37.990131 21869 webserver.cc:460] Webserver started at http://127.16.202.1:33635/ using document root <none> and password file <none>
I20260925 00:53:37.990695 21869 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:37.990762 21869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:37.990955 21869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:37.993124 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "601408dd21c14161bf41decb0459766b"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "c5b1b49b87a96ade53a887a17e791033"
server_key_iv: "1d410751f45ead628d8d1e79f8b9f1e5"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.993677 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "601408dd21c14161bf41decb0459766b"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "c5b1b49b87a96ade53a887a17e791033"
server_key_iv: "1d410751f45ead628d8d1e79f8b9f1e5"
server_key_version: "encryptionkey@0"
I20260925 00:53:37.997491 21869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:38.000113 21883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.001370 21869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:38.001490 21869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "601408dd21c14161bf41decb0459766b"
format_stamp: "Formatted at 2026-09-25 00:53:37 on dist-test-slave-rpk1"
server_key: "c5b1b49b87a96ade53a887a17e791033"
server_key_iv: "1d410751f45ead628d8d1e79f8b9f1e5"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.001600 21869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:38.006177 21869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:38.006799 21869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:38.006958 21869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:38.007503 21869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:38.008397 21869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:38.008441 21869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.008492 21869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:38.008520 21869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.016952 21869 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43639
I20260925 00:53:38.017014 21996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43639 every 8 connection(s)
I20260925 00:53:38.017920 21869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:38.019985 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 21869
I20260925 00:53:38.020128 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:38.020426 17192 external_mini_cluster.cc:1428] Setting key ef9b9eb1ad8340f47982ad8b54533a19
I20260925 00:53:38.022496 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33019
--builtin_ntp_servers=127.16.202.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 00:53:38.024996 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.019903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:48833 (local address 127.16.202.62:33019)
0925 00:53:38.020060 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.020065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.021856 (+ 1791us) server_negotiation.cc:407] Connection header received
0925 00:53:38.023203 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.023206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.023249 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.023329 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.024543 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.024546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.024549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.024560 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.024582 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.024589 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.024667 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.024833 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.024873 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:38.026134 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.020426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33019 (local address 127.16.202.1:48833)
0925 00:53:38.021698 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.021744 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.022989 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.023511 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.023518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.023642 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.024215 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.024218 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.024276 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.024368 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.024666 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.024693 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.024698 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.024886 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.025152 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260925 00:53:38.027405 21997 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33019
I20260925 00:53:38.027635 21997 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:38.028049 21997 heartbeater.cc:511] Master 127.16.202.62:33019 requested a full tablet report, sending...
I20260925 00:53:38.029561 21809 ts_manager.cc:194] Registered new tserver with Master: 601408dd21c14161bf41decb0459766b (127.16.202.1:43639)
W20260925 00:53:38.131060 22000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:38.131352 22000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:38.131407 22000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:38.135386 22000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:38.135455 22000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:38.135531 22000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:38.140327 22000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:38.141422 22000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:38.142786 22000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:38.152276 22005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:38.152304 22006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:38.152453 22000 server_base.cc:1034] running on GCE node
W20260925 00:53:38.152276 22008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:38.152997 22000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:38.153669 22000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:38.154845 22000 hybrid_clock.cc:648] HybridClock initialized: now 1790297618154822 us; error 40 us; skew 500 ppm
I20260925 00:53:38.156788 22000 webserver.cc:460] Webserver started at http://127.16.202.2:43621/ using document root <none> and password file <none>
I20260925 00:53:38.157383 22000 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:38.157451 22000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:38.157657 22000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:38.159811 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "47a846a7da804dc7b61d3ae27be8ec8a"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "9cad3cf222e571d430f7da4a078d441e"
server_key_iv: "020b27c7691839b28effe494a4a7015f"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.160369 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "47a846a7da804dc7b61d3ae27be8ec8a"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "9cad3cf222e571d430f7da4a078d441e"
server_key_iv: "020b27c7691839b28effe494a4a7015f"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.164101 22000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:38.166947 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.168272 22000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:38.168437 22000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "47a846a7da804dc7b61d3ae27be8ec8a"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "9cad3cf222e571d430f7da4a078d441e"
server_key_iv: "020b27c7691839b28effe494a4a7015f"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.168551 22000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:38.173038 22000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:38.173640 22000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:38.173805 22000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:38.174304 22000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:38.175187 22000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:38.175233 22000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.175282 22000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:38.175309 22000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.184037 22127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:33181 every 8 connection(s)
I20260925 00:53:38.184015 22000 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:33181
I20260925 00:53:38.185140 22000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:38.189106 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22000
I20260925 00:53:38.189239 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:38.189587 17192 external_mini_cluster.cc:1428] Setting key b68716d808cf5bfe1addf0602da76e34
I20260925 00:53:38.191490 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33019
--builtin_ntp_servers=127.16.202.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 00:53:38.191982 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.187047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:49971 (local address 127.16.202.62:33019)
0925 00:53:38.187249 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.187252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.188465 (+ 1213us) server_negotiation.cc:407] Connection header received
0925 00:53:38.190237 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.190240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.190287 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.190355 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.191479 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.191483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.191485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.191498 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.191522 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.191532 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.191644 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.191806 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.191852 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 00:53:38.193054 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.187447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33019 (local address 127.16.202.2:49971)
0925 00:53:38.188265 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.188309 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.189999 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.190548 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.190556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.190653 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.191208 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.191210 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.191252 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.191327 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.191656 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.191666 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.191673 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.191844 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.192116 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":124,"threads_started":1}
I20260925 00:53:38.194242 22128 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33019
I20260925 00:53:38.194489 22128 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:38.194888 22128 heartbeater.cc:511] Master 127.16.202.62:33019 requested a full tablet report, sending...
I20260925 00:53:38.195951 21809 ts_manager.cc:194] Registered new tserver with Master: 47a846a7da804dc7b61d3ae27be8ec8a (127.16.202.2:33181)
W20260925 00:53:38.303629 22131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:38.303907 22131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:38.303947 22131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:38.308081 22131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:38.308142 22131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:38.308202 22131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:38.313083 22131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:38.314108 22131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:38.315462 22131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:38.324935 22136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:38.324927 22137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:38.324941 22139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:38.325289 22131 server_base.cc:1034] running on GCE node
I20260925 00:53:38.325619 22131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:38.326195 22131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:38.327361 22131 hybrid_clock.cc:648] HybridClock initialized: now 1790297618327347 us; error 37 us; skew 500 ppm
I20260925 00:53:38.329064 22131 webserver.cc:460] Webserver started at http://127.16.202.3:42827/ using document root <none> and password file <none>
I20260925 00:53:38.329643 22131 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:38.329705 22131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:38.329888 22131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:38.331861 22131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "a49171305f1e47ecb29bcc768a6ab42f"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "fe7988af4e9653ace526ffb8f0c1bb4d"
server_key_iv: "856edb8ee314f7143c5c80802be47b4a"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.332386 22131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "a49171305f1e47ecb29bcc768a6ab42f"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "fe7988af4e9653ace526ffb8f0c1bb4d"
server_key_iv: "856edb8ee314f7143c5c80802be47b4a"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.335860 22131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:53:38.338301 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.339556 22131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:38.339661 22131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "a49171305f1e47ecb29bcc768a6ab42f"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "fe7988af4e9653ace526ffb8f0c1bb4d"
server_key_iv: "856edb8ee314f7143c5c80802be47b4a"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.339766 22131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:38.344318 22131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:38.345010 22131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:38.345170 22131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:38.345712 22131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:38.346602 22131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:38.346647 22131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.346697 22131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:38.346727 22131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.354794 22131 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:38167
I20260925 00:53:38.354823 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:38167 every 8 connection(s)
I20260925 00:53:38.355691 22131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:38.357990 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22131
I20260925 00:53:38.358088 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:38.358474 17192 external_mini_cluster.cc:1428] Setting key d453a28564bc7986cf0cd592daeb9167
I20260925 00:53:38.362196 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.357583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:36241 (local address 127.16.202.62:33019)
0925 00:53:38.357736 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.357740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.359067 (+ 1327us) server_negotiation.cc:407] Connection header received
0925 00:53:38.360591 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.360595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.360638 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.360769 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.361832 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.361835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.361837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.361848 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.361869 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.361877 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.361959 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.362053 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.362090 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:53:38.363101 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.358017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33019 (local address 127.16.202.3:36241)
0925 00:53:38.358870 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.358914 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.360366 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.360948 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.360955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.361053 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.361574 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.361576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.361618 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.361690 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.361950 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.361955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.361958 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.362085 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.362298 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":124,"threads_started":1}
I20260925 00:53:38.364212 22259 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33019
I20260925 00:53:38.364485 22259 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:38.364971 22259 heartbeater.cc:511] Master 127.16.202.62:33019 requested a full tablet report, sending...
I20260925 00:53:38.366027 21809 ts_manager.cc:194] Registered new tserver with Master: a49171305f1e47ecb29bcc768a6ab42f (127.16.202.3:38167)
I20260925 00:53:38.372714 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:53:38.376245 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.374667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.16.202.62:33019)
0925 00:53:38.374829 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.374834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.374961 (+ 127us) server_negotiation.cc:407] Connection header received
0925 00:53:38.375097 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.375099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.375135 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.375194 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.375870 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.375874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.375877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.375888 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.375907 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.375915 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.375988 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.376073 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.376111 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 00:53:38.379040 21809 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:38.382230 21809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:38.395301 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.391908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:38167 (local address 127.0.0.1:57620)
0925 00:53:38.392509 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.392563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.392667 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.393418 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.393424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.393511 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.394084 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.394086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.394107 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.394180 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.394935 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.394944 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.394950 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.395081 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.395151 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":166,"threads_started":1}
I20260925 00:53:38.395351 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.392192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.16.202.3:38167)
0925 00:53:38.392960 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.392971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.392996 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:53:38.393048 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.393055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.393251 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.393829 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.394696 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.394702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.394714 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.394724 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.394764 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.394791 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.394952 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.395117 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.395186 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":100,"threads_started":1}
I20260925 00:53:38.395457 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.392808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43639 (local address 127.0.0.1:43500)
0925 00:53:38.393302 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.393317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.393396 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.394499 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.394502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.394514 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.394891 (+ 377us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.394893 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.394904 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.394932 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.395226 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.395229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.395232 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.395308 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.395342 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":91,"threads_started":1}
I20260925 00:53:38.395944 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.393133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.16.202.1:43639)
0925 00:53:38.393547 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.393558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.393582 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:53:38.393639 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.393645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.393845 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.393978 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.395039 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.395045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.395058 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.395071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.395115 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.395145 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.395237 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.395319 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.395793 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":128,"threads_started":1}
I20260925 00:53:38.396140 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.392139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33181 (local address 127.0.0.1:37350)
0925 00:53:38.392745 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.392757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.392817 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.394702 (+ 1885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.394705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.394720 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.395503 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.395505 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.395514 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.395538 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.395927 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.395930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.395933 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.396001 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.396033 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":94,"threads_started":1}
I20260925 00:53:38.396549 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.392548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.16.202.2:33181)
0925 00:53:38.393676 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.393688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.393712 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:53:38.393763 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.393768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.393945 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.394082 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.395651 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.395656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.395667 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.395679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.395749 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.395787 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.395885 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.396335 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.396407 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":172,"threads_started":1}
I20260925 00:53:38.398847 21931 tablet_service.cc:1467] Processing CreateTablet for tablet f71dd3561d874b139611fa0bd05365cb (DEFAULT_TABLE table=test-table [id=0d7cf301c86745bbb6463515889aea26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:38.398846 22193 tablet_service.cc:1467] Processing CreateTablet for tablet f71dd3561d874b139611fa0bd05365cb (DEFAULT_TABLE table=test-table [id=0d7cf301c86745bbb6463515889aea26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:38.398831 22062 tablet_service.cc:1467] Processing CreateTablet for tablet f71dd3561d874b139611fa0bd05365cb (DEFAULT_TABLE table=test-table [id=0d7cf301c86745bbb6463515889aea26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:38.399959 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71dd3561d874b139611fa0bd05365cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:38.399960 22062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71dd3561d874b139611fa0bd05365cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:38.399959 21931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71dd3561d874b139611fa0bd05365cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:38.407047 22276 tablet_bootstrap.cc:492] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Bootstrap starting.
I20260925 00:53:38.408115 22277 tablet_bootstrap.cc:492] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: Bootstrap starting.
I20260925 00:53:38.408999 22278 tablet_bootstrap.cc:492] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Bootstrap starting.
I20260925 00:53:38.409832 22276 tablet_bootstrap.cc:654] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:38.410132 22277 tablet_bootstrap.cc:654] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:38.410784 22276 log.cc:824] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:38.410816 22277 log.cc:824] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:38.411164 22278 tablet_bootstrap.cc:654] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:38.411911 22278 log.cc:824] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:38.413115 22277 tablet_bootstrap.cc:492] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: No bootstrap required, opened a new log
I20260925 00:53:38.413328 22277 ts_tablet_manager.cc:1397] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:53:38.413923 22278 tablet_bootstrap.cc:492] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: No bootstrap required, opened a new log
I20260925 00:53:38.414114 22278 ts_tablet_manager.cc:1397] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:53:38.418002 22277 raft_consensus.cc:348] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.418313 22277 raft_consensus.cc:374] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:38.418370 22277 raft_consensus.cc:729] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601408dd21c14161bf41decb0459766b, State: Initialized, Role: FOLLOWER
I20260925 00:53:38.418419 22278 raft_consensus.cc:348] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.418651 22278 raft_consensus.cc:374] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:38.418713 22278 raft_consensus.cc:729] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49171305f1e47ecb29bcc768a6ab42f, State: Initialized, Role: FOLLOWER
I20260925 00:53:38.418927 22277 consensus_queue.cc:260] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.419219 22278 consensus_queue.cc:260] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.419788 21997 heartbeater.cc:503] Master 127.16.202.62:33019 was elected leader, sending a full tablet report...
I20260925 00:53:38.420053 22277 ts_tablet_manager.cc:1428] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 00:53:38.420097 22278 ts_tablet_manager.cc:1428] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260925 00:53:38.420167 22276 tablet_bootstrap.cc:492] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: No bootstrap required, opened a new log
I20260925 00:53:38.420374 22259 heartbeater.cc:503] Master 127.16.202.62:33019 was elected leader, sending a full tablet report...
I20260925 00:53:38.420463 22276 ts_tablet_manager.cc:1397] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260925 00:53:38.425207 22276 raft_consensus.cc:348] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.425510 22276 raft_consensus.cc:374] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:38.425585 22276 raft_consensus.cc:729] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a846a7da804dc7b61d3ae27be8ec8a, State: Initialized, Role: FOLLOWER
I20260925 00:53:38.425995 22276 consensus_queue.cc:260] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.426637 22128 heartbeater.cc:503] Master 127.16.202.62:33019 was elected leader, sending a full tablet report...
I20260925 00:53:38.426930 22276 ts_tablet_manager.cc:1428] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260925 00:53:38.436080 22129 tablet.cc:2369] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:38.494159 22282 raft_consensus.cc:482] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:38.494391 22282 raft_consensus.cc:504] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.495558 22282 leader_election.cc:288] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a49171305f1e47ecb29bcc768a6ab42f (127.16.202.3:38167), 47a846a7da804dc7b61d3ae27be8ec8a (127.16.202.2:33181)
I20260925 00:53:38.497525 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.495826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33181 (local address 127.16.202.1:50693)
0925 00:53:38.495933 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.495949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.496029 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.496318 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.496322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.496339 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.496922 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.496924 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.496937 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.496965 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.497246 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.497249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.497253 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.497348 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.497388 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 00:53:38.497556 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.495922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:50693 (local address 127.16.202.2:33181)
0925 00:53:38.496049 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.496054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.496069 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:38.496125 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.496128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.496161 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.496301 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.497107 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.497110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.497113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.497125 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.497148 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.497160 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.497251 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.497377 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.497423 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:53:38.497928 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.495966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:38823 (local address 127.16.202.3:38167)
0925 00:53:38.496079 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.496082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.496590 (+ 508us) server_negotiation.cc:407] Connection header received
0925 00:53:38.496646 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.496649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.496717 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.496799 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.497463 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.497467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.497470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.497482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.497502 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.497512 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.497599 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.497757 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.497802 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 00:53:38.498308 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.495826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:38167 (local address 127.16.202.1:38823)
0925 00:53:38.496460 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.496474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.496573 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.496854 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.496857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.496873 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:38.497299 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:53:38.497301 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:38.497313 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:53:38.497338 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:38.497595 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:38.497598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:38.497601 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:53:38.498134 (+ 533us) client_negotiation.cc:240] Negotiation successful
0925 00:53:38.498173 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":265,"thread_start_us":132,"threads_started":1}
I20260925 00:53:38.498417 22082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71dd3561d874b139611fa0bd05365cb" candidate_uuid: "601408dd21c14161bf41decb0459766b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" is_pre_election: true
I20260925 00:53:38.498745 22082 raft_consensus.cc:2454] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601408dd21c14161bf41decb0459766b in term 0.
I20260925 00:53:38.498891 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71dd3561d874b139611fa0bd05365cb" candidate_uuid: "601408dd21c14161bf41decb0459766b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49171305f1e47ecb29bcc768a6ab42f" is_pre_election: true
I20260925 00:53:38.499183 22213 raft_consensus.cc:2454] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601408dd21c14161bf41decb0459766b in term 0.
I20260925 00:53:38.499210 21887 leader_election.cc:302] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a846a7da804dc7b61d3ae27be8ec8a, 601408dd21c14161bf41decb0459766b; no voters:
I20260925 00:53:38.499467 22282 raft_consensus.cc:2781] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:38.499552 22282 raft_consensus.cc:482] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:38.499589 22282 raft_consensus.cc:3037] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:38.500597 22282 raft_consensus.cc:504] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.500949 22282 leader_election.cc:288] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [CANDIDATE]: Term 1 election: Requested vote from peers a49171305f1e47ecb29bcc768a6ab42f (127.16.202.3:38167), 47a846a7da804dc7b61d3ae27be8ec8a (127.16.202.2:33181)
I20260925 00:53:38.501296 22082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71dd3561d874b139611fa0bd05365cb" candidate_uuid: "601408dd21c14161bf41decb0459766b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a846a7da804dc7b61d3ae27be8ec8a"
I20260925 00:53:38.501335 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71dd3561d874b139611fa0bd05365cb" candidate_uuid: "601408dd21c14161bf41decb0459766b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49171305f1e47ecb29bcc768a6ab42f"
I20260925 00:53:38.501417 22082 raft_consensus.cc:3037] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:38.501498 22213 raft_consensus.cc:3037] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:38.502421 22082 raft_consensus.cc:2454] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601408dd21c14161bf41decb0459766b in term 1.
I20260925 00:53:38.502597 22213 raft_consensus.cc:2454] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601408dd21c14161bf41decb0459766b in term 1.
I20260925 00:53:38.502707 21887 leader_election.cc:302] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a846a7da804dc7b61d3ae27be8ec8a, 601408dd21c14161bf41decb0459766b; no voters:
I20260925 00:53:38.502887 22282 raft_consensus.cc:2781] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:38.503082 22282 raft_consensus.cc:686] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [term 1 LEADER]: Becoming Leader. State: Replica: 601408dd21c14161bf41decb0459766b, State: Running, Role: LEADER
I20260925 00:53:38.503449 22282 consensus_queue.cc:237] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } }
I20260925 00:53:38.506726 21807 catalog_manager.cc:5495] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b reported cstate change: term changed from 0 to 1, leader changed from <none> to 601408dd21c14161bf41decb0459766b (127.16.202.1). New cstate: current_term: 1 leader_uuid: "601408dd21c14161bf41decb0459766b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "601408dd21c14161bf41decb0459766b" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43639 } health_report { overall_health: HEALTHY } } }
W20260925 00:53:38.519109 21998 tablet.cc:2369] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:38.519902 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.518187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.16.202.1:43639)
0925 00:53:38.518317 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.518355 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.518370 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:38.518538 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.518540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.518604 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.518672 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.519491 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.519495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.519498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.519511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.519531 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.519540 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.519626 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.519727 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.519768 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:53:38.525630 22213 raft_consensus.cc:1261] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 1 FOLLOWER]: Refusing update from remote peer 601408dd21c14161bf41decb0459766b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:38.525630 22082 raft_consensus.cc:1261] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 1 FOLLOWER]: Refusing update from remote peer 601408dd21c14161bf41decb0459766b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:38.526315 22286 consensus_queue.cc:1035] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49171305f1e47ecb29bcc768a6ab42f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:38.526476 22282 consensus_queue.cc:1035] T f71dd3561d874b139611fa0bd05365cb P 601408dd21c14161bf41decb0459766b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a846a7da804dc7b61d3ae27be8ec8a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:38.536352 22290 mvcc.cc:205] Tried to move back new op lower bound from 7333059045474037760 to 7333059045392535552. Current Snapshot: MvccSnapshot[applied={T|T < 7333059045474037760}]
I20260925 00:53:38.541527 21809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:38.541695 21809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:38.544198 21809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f2c443923d0493894a2e2de656d4fb1: Sending DeleteTablet for 3 replicas of tablet f71dd3561d874b139611fa0bd05365cb
I20260925 00:53:38.545083 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet f71dd3561d874b139611fa0bd05365cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:38 UTC) from {username='slave'} at 127.0.0.1:57620
I20260925 00:53:38.545147 22062 tablet_service.cc:1514] Processing DeleteTablet for tablet f71dd3561d874b139611fa0bd05365cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:38 UTC) from {username='slave'} at 127.0.0.1:37350
I20260925 00:53:38.545658 22300 tablet_replica.cc:330] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: stopping tablet replica
I20260925 00:53:38.545674 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21869
I20260925 00:53:38.546016 22300 raft_consensus.cc:2229] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:38.546418 22300 raft_consensus.cc:2258] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:38.546712 22299 tablet_replica.cc:330] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: stopping tablet replica
I20260925 00:53:38.547056 22299 raft_consensus.cc:2229] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:38.547396 22299 raft_consensus.cc:2258] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:38.548106 22300 ts_tablet_manager.cc:1905] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:38.548857 22299 ts_tablet_manager.cc:1905] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:38.550858 22300 ts_tablet_manager.cc:1918] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:38.550880 22299 ts_tablet_manager.cc:1918] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:38.550958 22300 log.cc:1196] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/f71dd3561d874b139611fa0bd05365cb
I20260925 00:53:38.550958 22299 log.cc:1196] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/f71dd3561d874b139611fa0bd05365cb
I20260925 00:53:38.551242 22299 ts_tablet_manager.cc:1939] T f71dd3561d874b139611fa0bd05365cb P a49171305f1e47ecb29bcc768a6ab42f: Deleting consensus metadata
I20260925 00:53:38.551259 22300 ts_tablet_manager.cc:1939] T f71dd3561d874b139611fa0bd05365cb P 47a846a7da804dc7b61d3ae27be8ec8a: Deleting consensus metadata
I20260925 00:53:38.552371 21793 catalog_manager.cc:4841] TS a49171305f1e47ecb29bcc768a6ab42f (127.16.202.3:38167): tablet f71dd3561d874b139611fa0bd05365cb (table test-table [id=0d7cf301c86745bbb6463515889aea26]) successfully deleted
I20260925 00:53:38.552371 21796 catalog_manager.cc:4841] TS 47a846a7da804dc7b61d3ae27be8ec8a (127.16.202.2:33181): tablet f71dd3561d874b139611fa0bd05365cb (table test-table [id=0d7cf301c86745bbb6463515889aea26]) successfully deleted
I20260925 00:53:38.553357 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22000
W20260925 00:53:38.553512 21793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:43639 (error 108)
I20260925 00:53:38.554883 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.554612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43639 (local address 127.0.0.1:43504)
0925 00:53:38.554733 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.554803 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 00:53:38.555099 21793 catalog_manager.cc:4584] TS 601408dd21c14161bf41decb0459766b (127.16.202.1:43639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f71dd3561d874b139611fa0bd05365cb: Network error: Client connection negotiation failed: client connection to 127.16.202.1:43639: connect: Connection refused (error 111)
I20260925 00:53:38.559528 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22131
I20260925 00:53:38.564069 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 21778
2026-09-25T00:53:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (960 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 00:53:38.584283 17192 mini_oidc.cc:152] Starting JWKS server
I20260925 00:53:38.586042 17192 webserver.cc:460] Webserver started at https://127.0.0.1:40525/ using document root <none> and password file <none>
I20260925 00:53:38.586218 17192 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 00:53:38.586479 17192 webserver.cc:460] Webserver started at http://127.0.0.1:42479/ using document root <none> and password file <none>
2026-09-25T00:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:38Z Disabled control of system clock
I20260925 00:53:38.595233 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:46637
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:46637
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40525/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/testchainca.cert with env {}
W20260925 00:53:38.709117 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:38.709408 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:38.709450 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:38.713523 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:38.713594 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:38.713608 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:38.713624 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:38.713639 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:53:38.713654 22307 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:53:38.713667 22307 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40525/jwks.json
I20260925 00:53:38.719260 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:46637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:46637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40525/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:38.720424 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:38.721506 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:38.729907 22307 server_base.cc:1034] running on GCE node
W20260925 00:53:38.729871 22312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:38.729853 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:38.730057 22313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:38.730640 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:38.731714 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:38.732872 22307 hybrid_clock.cc:648] HybridClock initialized: now 1790297618732853 us; error 28 us; skew 500 ppm
I20260925 00:53:38.734675 22307 webserver.cc:460] Webserver started at http://127.16.202.62:44145/ using document root <none> and password file <none>
I20260925 00:53:38.735208 22307 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:38.735282 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:38.735482 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:38.737558 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "e0be3b7d526f46f5b0c8aac5569db0cb"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "847c1958f5a3077a649e05a8feda9a88"
server_key_iv: "d89b8b250b80f36c16633e17fd1bb009"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.738101 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "e0be3b7d526f46f5b0c8aac5569db0cb"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "847c1958f5a3077a649e05a8feda9a88"
server_key_iv: "d89b8b250b80f36c16633e17fd1bb009"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.741782 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:53:38.744130 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:38.745424 22307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:38.745535 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "e0be3b7d526f46f5b0c8aac5569db0cb"
format_stamp: "Formatted at 2026-09-25 00:53:38 on dist-test-slave-rpk1"
server_key: "847c1958f5a3077a649e05a8feda9a88"
server_key_iv: "d89b8b250b80f36c16633e17fd1bb009"
server_key_version: "encryptionkey@0"
I20260925 00:53:38.745630 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:38.767532 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:38.768296 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:38.768469 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:38.775175 22307 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:46637
I20260925 00:53:38.775188 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:46637 every 8 connection(s)
I20260925 00:53:38.776122 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:38.779316 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:38.781072 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22307
I20260925 00:53:38.781178 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:38.781477 17192 external_mini_cluster.cc:1428] Setting key ae563372df892d504eb42f82d4f0b0a2
I20260925 00:53:38.785950 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb: Bootstrap starting.
I20260925 00:53:38.788152 22374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:38.789047 22374 log.cc:824] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:38.790736 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.782928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.16.202.62:46637)
0925 00:53:38.783616 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.783632 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.783679 (+ 47us) server_negotiation.cc:407] Connection header received
0925 00:53:38.784342 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.784361 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.784801 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.785129 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:38.785553 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:38.786599 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:38.787462 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:38.787951 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:38.788090 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:38.788930 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:38.788937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:38.788953 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:38.789448 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:38.789503 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:38.789524 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:38.789609 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:38.789743 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 00:53:38.789999 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260925 00:53:38.790979 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb: No bootstrap required, opened a new log
I20260925 00:53:38.793618 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:53:38.794997 22374 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46637 } }
I20260925 00:53:38.795270 22374 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:38.795329 22374 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0be3b7d526f46f5b0c8aac5569db0cb, State: Initialized, Role: FOLLOWER
I20260925 00:53:38.795749 22374 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46637 } }
I20260925 00:53:38.795886 22374 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:38.795953 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:38.796049 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:38.797152 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46637 } }
I20260925 00:53:38.797484 22374 leader_election.cc:302] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0be3b7d526f46f5b0c8aac5569db0cb; no voters:
I20260925 00:53:38.797786 22374 leader_election.cc:288] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:38.797899 22379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:38.798110 22379 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [term 1 LEADER]: Becoming Leader. State: Replica: e0be3b7d526f46f5b0c8aac5569db0cb, State: Running, Role: LEADER
I20260925 00:53:38.798408 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46637 } }
I20260925 00:53:38.798985 22374 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:38.799957 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0be3b7d526f46f5b0c8aac5569db0cb. Latest consensus state: current_term: 1 leader_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46637 } } }
I20260925 00:53:38.800079 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:38.800433 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be3b7d526f46f5b0c8aac5569db0cb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 46637 } } }
I20260925 00:53:38.800530 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb [sys.catalog]: This master's current role is: LEADER
W20260925 00:53:38.803659 22394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:53:38.803740 22394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:53:38.803819 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:38.804526 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:38.810385 22388 catalog_manager.cc:1347] Generated new cluster ID: 746cf2e265864a08b9489d2503457b8a
I20260925 00:53:38.810462 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:38.815896 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:38.816890 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:38.826423 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0be3b7d526f46f5b0c8aac5569db0cb: Generated new TSK 0
I20260925 00:53:38.827160 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 00:53:38.870008 22404 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 00:53:38.870143 22404 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 00:53:38.870613 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:38.845946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.16.202.62:46637)
0925 00:53:38.846092 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:38.846097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:38.846309 (+ 212us) server_negotiation.cc:407] Connection header received
0925 00:53:38.846572 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:38.846575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:38.846629 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:38.846725 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:53:38.847020 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:38.847617 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:38.848878 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:38.849177 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:38.849261 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:38.849673 (+ 412us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:53:38.870136 (+ 20463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 00:53:38.870498 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 00:53:38.870671 22377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 00:53:38.872079 22403 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:38.845859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:46637 (local address 127.0.0.1:41698)
0925 00:53:38.846210 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:38.846225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:38.846369 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:38.846760 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:38.846763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:38.846772 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:53:38.846888 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:38.846895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:38.847728 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:38.847732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:38.848757 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:38.848770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:38.849298 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:38.849303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:38.849397 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:38.849461 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:53:38.870467 (+ 21006us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 00:53:38.870731 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.202.62:46637: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.202.62:46637: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 00:53:38.872869 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22307
2026-09-25T00:53:38Z chronyd exiting
I20260925 00:53:38.910028 17192 test_util.cc:175] -----------------------------------------------
I20260925 00:53:38.910120 17192 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297591180183-17192-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 00:53:38.911150 17192 mini_oidc.cc:152] Starting JWKS server
I20260925 00:53:38.912007 17192 webserver.cc:460] Webserver started at https://127.0.0.1:35771/ using document root <none> and password file <none>
I20260925 00:53:38.912096 17192 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 00:53:38.912309 17192 webserver.cc:460] Webserver started at http://127.0.0.1:39251/ using document root <none> and password file <none>
2026-09-25T00:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:38Z Disabled control of system clock
I20260925 00:53:38.921072 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:42741
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:42741
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/testchainca.cert with env {}
W20260925 00:53:39.031917 22410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:39.032220 22410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:39.032261 22410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:39.036744 22410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:39.036839 22410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:39.036856 22410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:39.036876 22410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:39.036895 22410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:53:39.036912 22410 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:53:39.036928 22410 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35771/jwks.json
I20260925 00:53:39.042703 22410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:42741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:39.043854 22410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:39.044896 22410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:39.053475 22418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:39.053483 22416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:39.053483 22415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:39.053961 22410 server_base.cc:1034] running on GCE node
I20260925 00:53:39.054538 22410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:39.055634 22410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:39.056764 22410 hybrid_clock.cc:648] HybridClock initialized: now 1790297619056746 us; error 36 us; skew 500 ppm
I20260925 00:53:39.058679 22410 webserver.cc:460] Webserver started at http://127.16.202.62:42025/ using document root <none> and password file <none>
I20260925 00:53:39.059230 22410 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:39.059290 22410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:39.059456 22410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:39.061578 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "c0acaedb4f524b7c8a13912a3d82eb54"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "34aebf9bf701fe1d666fcc698dc6f80a"
server_key_iv: "9d884f461cdddb3fe070915ec31ee2f0"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.062094 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "c0acaedb4f524b7c8a13912a3d82eb54"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "34aebf9bf701fe1d666fcc698dc6f80a"
server_key_iv: "9d884f461cdddb3fe070915ec31ee2f0"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.065656 22410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:53:39.068210 22424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:39.069550 22410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:39.069669 22410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "c0acaedb4f524b7c8a13912a3d82eb54"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "34aebf9bf701fe1d666fcc698dc6f80a"
server_key_iv: "9d884f461cdddb3fe070915ec31ee2f0"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.069764 22410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:39.091584 22410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:39.092301 22410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:39.092458 22410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:39.099581 22410 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:42741
I20260925 00:53:39.099599 22476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:42741 every 8 connection(s)
I20260925 00:53:39.100533 22410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:39.103739 22477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:39.107055 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22410
I20260925 00:53:39.107170 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:39.107448 17192 external_mini_cluster.cc:1428] Setting key 1e8495b1dd2bd4374c45e643a7ecd220
I20260925 00:53:39.109716 22477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54: Bootstrap starting.
I20260925 00:53:39.112509 22477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:39.113426 22477 log.cc:824] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:39.116082 22477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54: No bootstrap required, opened a new log
I20260925 00:53:39.118170 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:39.108897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.16.202.62:42741)
0925 00:53:39.109641 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:39.109662 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:39.109705 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:53:39.110548 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:39.110570 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:39.111112 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:39.111481 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:39.111939 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.113468 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.114606 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.115342 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.115524 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.116374 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:39.116379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:39.116394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:39.116837 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:39.116901 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:39.116923 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:39.117017 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:39.117157 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 00:53:39.117430 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":144,"threads_started":1}
I20260925 00:53:39.119962 22477 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 42741 } }
I20260925 00:53:39.120227 22477 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:39.120307 22477 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0acaedb4f524b7c8a13912a3d82eb54, State: Initialized, Role: FOLLOWER
I20260925 00:53:39.120796 22477 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 42741 } }
I20260925 00:53:39.120957 22477 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:39.121018 22477 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:39.121112 22477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:39.121271 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:53:39.122084 22477 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 42741 } }
I20260925 00:53:39.122460 22477 leader_election.cc:302] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0acaedb4f524b7c8a13912a3d82eb54; no voters:
I20260925 00:53:39.122793 22477 leader_election.cc:288] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:39.122867 22482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:39.123054 22482 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [term 1 LEADER]: Becoming Leader. State: Replica: c0acaedb4f524b7c8a13912a3d82eb54, State: Running, Role: LEADER
I20260925 00:53:39.123348 22482 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 42741 } }
I20260925 00:53:39.124082 22477 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:39.125066 22484 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0acaedb4f524b7c8a13912a3d82eb54. Latest consensus state: current_term: 1 leader_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 42741 } } }
I20260925 00:53:39.125281 22483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acaedb4f524b7c8a13912a3d82eb54" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 42741 } } }
I20260925 00:53:39.125372 22484 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:39.125396 22483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:39.125872 22491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:39.128706 22491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:39.134302 22491 catalog_manager.cc:1347] Generated new cluster ID: 3ed8d49e996541c984ff2ed25c3a8bf4
I20260925 00:53:39.134397 22491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:39.152390 22491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:39.153429 22491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:39.159060 22491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0acaedb4f524b7c8a13912a3d82eb54: Generated new TSK 0
I20260925 00:53:39.159808 22491 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 00:53:39.192967 22507 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 00:53:39.193064 22507 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 00:53:39.193684 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:39.170532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.16.202.62:42741)
0925 00:53:39.170666 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:39.170670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:39.170731 (+ 61us) server_negotiation.cc:407] Connection header received
0925 00:53:39.170904 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:39.170907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:39.170956 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:39.171046 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:53:39.171270 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.171834 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.173052 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.173288 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.173374 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.173691 (+ 317us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:53:39.193314 (+ 19623us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 00:53:39.193558 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 00:53:39.193768 22480 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 00:53:39.193897 22506 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:39.170334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:42741 (local address 127.0.0.1:47974)
0925 00:53:39.170635 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:39.170650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:39.170781 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:39.171058 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:39.171061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:39.171068 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:53:39.171180 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:39.171186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.171934 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.171937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:39.172934 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:39.172945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.173393 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.173397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:39.173487 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.173516 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:53:39.193523 (+ 20007us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 00:53:39.193704 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.202.62:42741: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":109,"threads_started":1}
I20260925 00:53:39.194504 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22410
2026-09-25T00:53:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 00:53:39.232748 17192 mini_oidc.cc:152] Starting JWKS server
I20260925 00:53:39.233603 17192 webserver.cc:460] Webserver started at https://127.0.0.1:33465/ using document root <none> and password file <none>
I20260925 00:53:39.233687 17192 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 00:53:39.233896 17192 webserver.cc:460] Webserver started at http://127.0.0.1:40265/ using document root <none> and password file <none>
2026-09-25T00:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:39Z Disabled control of system clock
I20260925 00:53:39.242813 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34225
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:34225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33465/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 00:53:39.353449 22513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:39.353736 22513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:39.353776 22513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:39.357846 22513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:39.357909 22513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:39.357923 22513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:39.357939 22513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:39.357954 22513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:53:39.357969 22513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:53:39.357981 22513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33465/jwks.json
I20260925 00:53:39.362921 22513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:34225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33465/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:39.364018 22513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:39.365410 22513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:39.374408 22518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:39.374384 22519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:39.374357 22521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:39.374604 22513 server_base.cc:1034] running on GCE node
I20260925 00:53:39.375408 22513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:39.376644 22513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:39.377830 22513 hybrid_clock.cc:648] HybridClock initialized: now 1790297619377813 us; error 34 us; skew 500 ppm
I20260925 00:53:39.379834 22513 webserver.cc:460] Webserver started at http://127.16.202.62:45531/ using document root <none> and password file <none>
I20260925 00:53:39.380867 22513 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:39.380940 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:39.381155 22513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:39.383280 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "d882028594114962bb0ba2f8a7abe817"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "f154d789808284e15f7c3e539c79c9fc"
server_key_iv: "fcd44e1d28f7b18ebacaf730694e122d"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.383834 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "d882028594114962bb0ba2f8a7abe817"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "f154d789808284e15f7c3e539c79c9fc"
server_key_iv: "fcd44e1d28f7b18ebacaf730694e122d"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.388036 22513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:39.390770 22527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:39.392107 22513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:39.392227 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "d882028594114962bb0ba2f8a7abe817"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "f154d789808284e15f7c3e539c79c9fc"
server_key_iv: "fcd44e1d28f7b18ebacaf730694e122d"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.392386 22513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:39.408146 22513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:39.408905 22513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:39.409078 22513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:39.415658 22513 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:34225
I20260925 00:53:39.415679 22579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:34225 every 8 connection(s)
I20260925 00:53:39.416575 22513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:39.418908 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22513
I20260925 00:53:39.419030 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:39.419337 17192 external_mini_cluster.cc:1428] Setting key db7efda3aaa8aecb75561479b653e3d6
I20260925 00:53:39.420167 22580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:39.426633 22580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817: Bootstrap starting.
I20260925 00:53:39.428771 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:39.420934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.16.202.62:34225)
0925 00:53:39.421638 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:39.421657 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:39.421696 (+ 39us) server_negotiation.cc:407] Connection header received
0925 00:53:39.422362 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:39.422387 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:39.422807 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:39.423137 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:39.423606 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.424580 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.425513 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.426002 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.426150 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.426974 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:39.426979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:39.426993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:39.427395 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:39.427458 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:39.427479 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:39.427564 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:39.427708 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 00:53:39.427978 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260925 00:53:39.429029 22580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:39.429756 22580 log.cc:824] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:39.431779 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:53:39.431977 22580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817: No bootstrap required, opened a new log
I20260925 00:53:39.435861 22580 raft_consensus.cc:348] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882028594114962bb0ba2f8a7abe817" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34225 } }
I20260925 00:53:39.436110 22580 raft_consensus.cc:374] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:39.436170 22580 raft_consensus.cc:729] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d882028594114962bb0ba2f8a7abe817, State: Initialized, Role: FOLLOWER
I20260925 00:53:39.436590 22580 consensus_queue.cc:260] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882028594114962bb0ba2f8a7abe817" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34225 } }
I20260925 00:53:39.436739 22580 raft_consensus.cc:388] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:39.436807 22580 raft_consensus.cc:482] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:39.436903 22580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:39.437871 22580 raft_consensus.cc:504] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882028594114962bb0ba2f8a7abe817" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34225 } }
I20260925 00:53:39.438207 22580 leader_election.cc:302] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d882028594114962bb0ba2f8a7abe817; no voters:
I20260925 00:53:39.438519 22580 leader_election.cc:288] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:39.438661 22585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:39.438881 22585 raft_consensus.cc:686] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [term 1 LEADER]: Becoming Leader. State: Replica: d882028594114962bb0ba2f8a7abe817, State: Running, Role: LEADER
I20260925 00:53:39.439217 22585 consensus_queue.cc:237] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882028594114962bb0ba2f8a7abe817" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34225 } }
I20260925 00:53:39.439874 22580 sys_catalog.cc:564] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:39.440959 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d882028594114962bb0ba2f8a7abe817. Latest consensus state: current_term: 1 leader_uuid: "d882028594114962bb0ba2f8a7abe817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882028594114962bb0ba2f8a7abe817" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34225 } } }
I20260925 00:53:39.441098 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:39.441334 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d882028594114962bb0ba2f8a7abe817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d882028594114962bb0ba2f8a7abe817" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34225 } } }
I20260925 00:53:39.441422 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:39.441525 22594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:39.444329 22594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:39.450798 22594 catalog_manager.cc:1347] Generated new cluster ID: 2654ad2f46154892ada5f5ded5f4de11
I20260925 00:53:39.450892 22594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:39.474138 22594 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:39.475380 22594 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:39.485929 22594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d882028594114962bb0ba2f8a7abe817: Generated new TSK 0
I20260925 00:53:39.486819 22594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:53:39.509552 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:39.504902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.16.202.62:34225)
0925 00:53:39.505056 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:39.505060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:39.505169 (+ 109us) server_negotiation.cc:407] Connection header received
0925 00:53:39.505370 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:39.505373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:39.505420 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:39.505516 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:53:39.505750 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.506298 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.507193 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.507420 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.507507 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.507817 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:53:39.509190 (+ 1373us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33465/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 00:53:39.509407 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33465/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 00:53:39.509625 22583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33465/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 00:53:39.511363 22609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:39.504777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34225 (local address 127.0.0.1:46126)
0925 00:53:39.505083 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:39.505097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:39.505212 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:39.505525 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:39.505528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:39.505535 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:53:39.505648 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:39.505654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.506421 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.506425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:39.507079 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:39.507087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.507535 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.507539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:39.507624 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.507647 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:53:39.509348 (+ 1701us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33465/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 00:53:39.511179 (+ 1831us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.202.62:34225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33465/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":1621,"thread_start_us":106,"threads_started":1}
I20260925 00:53:39.512060 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22513
E20260925 00:53:39.522203 22610 webserver.cc:533] Webserver: sslize failed:
E20260925 00:53:39.522295 22610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T00:53:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T00:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:39Z Disabled control of system clock
I20260925 00:53:39.561386 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:37567
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:37567
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/testchainca.cert with env {}
W20260925 00:53:39.674525 22614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:39.674794 22614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:39.674835 22614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:39.678970 22614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:39.679030 22614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:39.679044 22614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:39.679060 22614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:39.679075 22614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:53:39.679092 22614 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:53:39.679119 22614 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 00:53:39.684664 22614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:37567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:39.685845 22614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:39.686897 22614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:39.695113 22622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:39.695117 22619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:39.695117 22620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:39.695506 22614 server_base.cc:1034] running on GCE node
I20260925 00:53:39.696045 22614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:39.697324 22614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:39.698494 22614 hybrid_clock.cc:648] HybridClock initialized: now 1790297619698458 us; error 62 us; skew 500 ppm
I20260925 00:53:39.703610 22614 webserver.cc:460] Webserver started at http://127.16.202.62:43515/ using document root <none> and password file <none>
I20260925 00:53:39.704241 22614 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:39.704313 22614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:39.704568 22614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:39.706768 22614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "9298fdf251fe4f1aa51c18fa78bd55f3"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "22c03f0ba357ce3b6cc7241272cddf50"
server_key_iv: "ed67a48258b31fcb39c05c38d17780fd"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.707306 22614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "9298fdf251fe4f1aa51c18fa78bd55f3"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "22c03f0ba357ce3b6cc7241272cddf50"
server_key_iv: "ed67a48258b31fcb39c05c38d17780fd"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.710992 22614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:53:39.713562 22628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:39.714864 22614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:39.714972 22614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "9298fdf251fe4f1aa51c18fa78bd55f3"
format_stamp: "Formatted at 2026-09-25 00:53:39 on dist-test-slave-rpk1"
server_key: "22c03f0ba357ce3b6cc7241272cddf50"
server_key_iv: "ed67a48258b31fcb39c05c38d17780fd"
server_key_version: "encryptionkey@0"
I20260925 00:53:39.715090 22614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:39.730154 22614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:39.730885 22614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:39.731063 22614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:39.737658 22614 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:37567
I20260925 00:53:39.737682 22680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:37567 every 8 connection(s)
I20260925 00:53:39.738596 22614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:39.741460 22681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:39.747326 22681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3: Bootstrap starting.
I20260925 00:53:39.747421 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22614
I20260925 00:53:39.747519 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:39.747810 17192 external_mini_cluster.cc:1428] Setting key 08ea1521897de41146ed0e3858e7f57a
I20260925 00:53:39.749984 22681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:39.750703 22681 log.cc:824] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:39.752534 22681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3: No bootstrap required, opened a new log
I20260925 00:53:39.756088 22681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37567 } }
I20260925 00:53:39.756332 22681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:39.756368 22681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9298fdf251fe4f1aa51c18fa78bd55f3, State: Initialized, Role: FOLLOWER
I20260925 00:53:39.756839 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:39.749088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.16.202.62:37567)
0925 00:53:39.749713 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:39.749726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:39.749762 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:53:39.750444 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:39.750464 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:39.750892 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:39.751206 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:39.751633 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.752571 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.753462 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.753945 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.754089 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.754899 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:39.754904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:39.754916 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:53:39.755295 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:39.755359 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:53:39.755380 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:39.755462 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:39.755596 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 00:53:39.755855 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
I20260925 00:53:39.756834 22681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37567 } }
I20260925 00:53:39.756966 22681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:39.757019 22681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:39.757109 22681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:39.758030 22681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37567 } }
I20260925 00:53:39.758384 22681 leader_election.cc:302] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9298fdf251fe4f1aa51c18fa78bd55f3; no voters:
I20260925 00:53:39.758731 22681 leader_election.cc:288] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:39.758801 22686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:39.759017 22686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [term 1 LEADER]: Becoming Leader. State: Replica: 9298fdf251fe4f1aa51c18fa78bd55f3, State: Running, Role: LEADER
I20260925 00:53:39.759338 22686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37567 } }
I20260925 00:53:39.759749 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:53:39.760052 22681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:39.761262 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9298fdf251fe4f1aa51c18fa78bd55f3. Latest consensus state: current_term: 1 leader_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37567 } } }
I20260925 00:53:39.761387 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:39.761767 22695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:39.761741 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298fdf251fe4f1aa51c18fa78bd55f3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37567 } } }
I20260925 00:53:39.761873 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:39.765163 22695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:39.771652 22695 catalog_manager.cc:1347] Generated new cluster ID: a9a4a9620ff74853abb0ba82d856ecc1
I20260925 00:53:39.771735 22695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:39.802163 22695 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:39.803261 22695 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:39.812274 22695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9298fdf251fe4f1aa51c18fa78bd55f3: Generated new TSK 0
I20260925 00:53:39.813051 22695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:53:39.877285 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:39.822368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.16.202.62:37567)
0925 00:53:39.822503 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:39.822508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:39.822617 (+ 109us) server_negotiation.cc:407] Connection header received
0925 00:53:39.822780 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:39.822783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:39.822868 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:39.822988 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:53:39.823195 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.824017 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.824934 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.825312 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.825411 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.825722 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:53:39.876740 (+ 51018us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 00:53:39.877121 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 00:53:39.877375 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:39.822256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:37567 (local address 127.0.0.1:47622)
0925 00:53:39.822528 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:39.822542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:39.822644 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:39.823000 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:39.823002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:39.823010 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:53:39.823103 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:39.823108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.824155 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.824158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:39.824827 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:39.824837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:39.825446 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:39.825449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:39.825522 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:39.825547 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:53:39.877030 (+ 51483us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 00:53:39.877218 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.202.62:37567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
W20260925 00:53:39.877390 22684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260925 00:53:39.878068 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22614
2026-09-25T00:53:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (343 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:40 dist-test-slave-rpk1 krb5kdc[22722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:40 dist-test-slave-rpk1 krb5kdc[22722](info): set up 2 sockets
Sep 25 00:53:40 dist-test-slave-rpk1 krb5kdc[22722](info): commencing operation
krb5kdc: starting...
W20260925 00:53:41.072901 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:41.196731 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45273
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:41.312249 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:41.312567 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:41.312611 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:41.316761 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:41.316851 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:41.316872 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:41.316895 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:41.316912 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:41.322547 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:41.323664 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:41.324724 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:41.333178 22735 server_base.cc:1034] running on GCE node
W20260925 00:53:41.333135 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:41.333148 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:41.333130 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:41.333940 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:41.335047 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:41.336282 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790297621336255 us; error 35 us; skew 500 ppm
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:41.339082 22735 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:41.339987 22735 webserver.cc:460] Webserver started at http://127.16.202.62:32841/ using document root <none> and password file <none>
I20260925 00:53:41.340528 22735 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:41.340590 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:41.340819 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:41.342906 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "e49f1d0ef09d4e369b347aa52fd33841"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "b788073bfb868f52dc9ee3f9ee973299"
server_key_iv: "76487fd6c28866738df3a6a0fdf9a103"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.343492 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "e49f1d0ef09d4e369b347aa52fd33841"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "b788073bfb868f52dc9ee3f9ee973299"
server_key_iv: "76487fd6c28866738df3a6a0fdf9a103"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.347472 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 00:53:41.350015 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.351351 22735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:41.351472 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "e49f1d0ef09d4e369b347aa52fd33841"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "b788073bfb868f52dc9ee3f9ee973299"
server_key_iv: "76487fd6c28866738df3a6a0fdf9a103"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.351575 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:41.363891 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:41.367261 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:41.367457 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:41.374611 22735 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:45273
I20260925 00:53:41.374639 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:45273 every 8 connection(s)
I20260925 00:53:41.375604 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:41.378449 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:41.383208 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22735
I20260925 00:53:41.383322 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:41.383579 17192 external_mini_cluster.cc:1428] Setting key 9da22d11d1aca578f6b4c9d3c4bd18b3
I20260925 00:53:41.384658 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841: Bootstrap starting.
I20260925 00:53:41.387503 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:41.388379 22803 log.cc:824] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:41.390676 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841: No bootstrap required, opened a new log
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:41.394831 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f1d0ef09d4e369b347aa52fd33841" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45273 } }
I20260925 00:53:41.395130 22803 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:41.395195 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49f1d0ef09d4e369b347aa52fd33841, State: Initialized, Role: FOLLOWER
I20260925 00:53:41.395807 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f1d0ef09d4e369b347aa52fd33841" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45273 } }
I20260925 00:53:41.395943 22803 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:41.396000 22803 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:41.396095 22803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:41.397030 22803 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f1d0ef09d4e369b347aa52fd33841" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45273 } }
I20260925 00:53:41.397347 22803 leader_election.cc:302] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49f1d0ef09d4e369b347aa52fd33841; no voters:
I20260925 00:53:41.397645 22803 leader_election.cc:288] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:41.397886 22808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:41.398108 22808 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [term 1 LEADER]: Becoming Leader. State: Replica: e49f1d0ef09d4e369b347aa52fd33841, State: Running, Role: LEADER
I20260925 00:53:41.398419 22808 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f1d0ef09d4e369b347aa52fd33841" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45273 } }
I20260925 00:53:41.398752 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:41.385078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.16.202.62:45273)
0925 00:53:41.385529 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:41.385548 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:41.385590 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:53:41.386449 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:41.386474 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:41.386922 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:41.387319 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:41.388127 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.389491 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.390410 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.391102 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.391281 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:41.393684 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:41.393691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:41.393711 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:41.393745 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:41.395633 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:41.396142 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:41.396149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:41.396156 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:41.396285 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:41.396567 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:41.396570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:41.396572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:41.397047 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:41.397229 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:41.397542 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 00:53:41.397865 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260925 00:53:41.399060 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:41.400621 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49f1d0ef09d4e369b347aa52fd33841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f1d0ef09d4e369b347aa52fd33841" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45273 } } }
I20260925 00:53:41.400621 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49f1d0ef09d4e369b347aa52fd33841. Latest consensus state: current_term: 1 leader_uuid: "e49f1d0ef09d4e369b347aa52fd33841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f1d0ef09d4e369b347aa52fd33841" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45273 } } }
I20260925 00:53:41.400861 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:41.400861 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:41.401548 22822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:41.404271 22822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:41.409673 22822 catalog_manager.cc:1347] Generated new cluster ID: 1e3172e996c6454685c3f9ea01b81afb
I20260925 00:53:41.409737 22822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:41.425124 22822 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:41.426457 22822 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:41.437047 22822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49f1d0ef09d4e369b347aa52fd33841: Generated new TSK 0
I20260925 00:53:41.437856 22822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:41.472784 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45273
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:41.593978 22829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:41.594468 22829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:41.594566 22829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:41.600528 22829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:41.600598 22829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:41.600663 22829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:41.606251 22829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:41.607312 22829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:41.608297 22829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:41.617749 22834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:41.617725 22837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:41.617769 22835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:41.617901 22829 server_base.cc:1034] running on GCE node
I20260925 00:53:41.618474 22829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:41.619166 22829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:41.620364 22829 hybrid_clock.cc:648] HybridClock initialized: now 1790297621620341 us; error 41 us; skew 500 ppm
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:41.623158 22829 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:41.624050 22829 webserver.cc:460] Webserver started at http://127.16.202.1:38919/ using document root <none> and password file <none>
I20260925 00:53:41.624603 22829 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:41.624691 22829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:41.624914 22829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:41.626962 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "3a3136628d1148f9b764c940be2e362f"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "90dccb8e34d5954af4909381c8658cd0"
server_key_iv: "5cf7d1cd8a352bd9812a9ef68a26e151"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.627527 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "3a3136628d1148f9b764c940be2e362f"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "90dccb8e34d5954af4909381c8658cd0"
server_key_iv: "5cf7d1cd8a352bd9812a9ef68a26e151"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.631232 22829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:53:41.633733 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.635165 22829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:41.635334 22829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "3a3136628d1148f9b764c940be2e362f"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "90dccb8e34d5954af4909381c8658cd0"
server_key_iv: "5cf7d1cd8a352bd9812a9ef68a26e151"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.635452 22829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:41.651098 22829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:41.654667 22829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:41.654870 22829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:41.655531 22829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:41.656410 22829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:41.656457 22829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.656497 22829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:41.656523 22829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.665127 22829 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:39245
I20260925 00:53:41.665150 22957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:39245 every 8 connection(s)
I20260925 00:53:41.666219 22829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:41.669973 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22829
I20260925 00:53:41.670080 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:41.670341 17192 external_mini_cluster.cc:1428] Setting key baf6e1a41effbf60debab9abe24fa6fa
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:41.683547 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:41.668190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:36073 (local address 127.16.202.62:45273)
0925 00:53:41.668387 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:41.668391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:41.669364 (+ 973us) server_negotiation.cc:407] Connection header received
0925 00:53:41.670814 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:41.670817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:41.670880 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:41.670975 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:41.673287 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.674055 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.676038 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.676300 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.676397 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:41.679669 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:41.679673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:41.679675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:41.679705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:41.681192 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:41.681768 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:41.681772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:41.681773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:41.681872 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:41.682302 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:41.682305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:41.682306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:41.682572 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:41.682653 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:41.683225 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 00:53:41.683372 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:41.684489 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:41.668649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45273 (local address 127.16.202.1:36073)
0925 00:53:41.669193 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:41.669241 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:41.670550 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:41.671192 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:41.671202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:41.672185 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:41.673095 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:41.673114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.674259 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.674263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:41.675821 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:41.675836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.676484 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.676489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:41.676790 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:41.677509 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:41.677533 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:41.679409 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:41.681324 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:41.681330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:41.681346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:41.681620 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:41.682010 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:41.682013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:41.682014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:41.682193 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:41.682663 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:41.682668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:41.682953 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 00:53:41.683132 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 00:53:41.683513 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:41.685952 22958 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45273
I20260925 00:53:41.686239 22958 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:41.686883 22958 heartbeater.cc:511] Master 127.16.202.62:45273 requested a full tablet report, sending...
I20260925 00:53:41.688759 22767 ts_manager.cc:194] Registered new tserver with Master: 3a3136628d1148f9b764c940be2e362f (127.16.202.1:39245)
I20260925 00:53:41.689944 22767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:36073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:41.700860 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45273
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:41.813086 22963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:41.813357 22963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:41.813400 22963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:41.817504 22963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:41.817565 22963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:41.817629 22963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:41.822969 22963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:41.824066 22963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:41.825141 22963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:41.835551 22971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:41.835552 22968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:41.835587 22969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:41.835736 22963 server_base.cc:1034] running on GCE node
I20260925 00:53:41.836475 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:41.837230 22963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:41.838428 22963 hybrid_clock.cc:648] HybridClock initialized: now 1790297621838410 us; error 38 us; skew 500 ppm
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:41.841378 22963 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:41.842306 22963 webserver.cc:460] Webserver started at http://127.16.202.2:41479/ using document root <none> and password file <none>
I20260925 00:53:41.842881 22963 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:41.842943 22963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:41.843114 22963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:41.845316 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "86e66345e8fa41ba96806fe6a71b8503"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "e52dd5c1d44dd81d303480d6630e6b5d"
server_key_iv: "cbc6b41b8664f929941c91c94a97912b"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.845873 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "86e66345e8fa41ba96806fe6a71b8503"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "e52dd5c1d44dd81d303480d6630e6b5d"
server_key_iv: "cbc6b41b8664f929941c91c94a97912b"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.849848 22963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:41.852505 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.853885 22963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:41.854001 22963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "86e66345e8fa41ba96806fe6a71b8503"
format_stamp: "Formatted at 2026-09-25 00:53:41 on dist-test-slave-rpk1"
server_key: "e52dd5c1d44dd81d303480d6630e6b5d"
server_key_iv: "cbc6b41b8664f929941c91c94a97912b"
server_key_version: "encryptionkey@0"
I20260925 00:53:41.854116 22963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:41.866165 22963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:41.869285 22963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:41.869462 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:41.870086 22963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:41.870896 22963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:41.870944 22963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.870981 22963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:41.871009 22963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:41.879117 22963 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:34053
I20260925 00:53:41.879148 23091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:34053 every 8 connection(s)
I20260925 00:53:41.879988 22963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:41.887378 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 22963
I20260925 00:53:41.887524 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:41.887842 17192 external_mini_cluster.cc:1428] Setting key cf07ffebfe67f2371a1eaafc49244177
Sep 25 00:53:41 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297621, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:41.893956 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:41.881764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:56813 (local address 127.16.202.62:45273)
0925 00:53:41.881928 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:41.881933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:41.882769 (+ 836us) server_negotiation.cc:407] Connection header received
0925 00:53:41.884107 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:41.884110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:41.884163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:41.884249 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:41.885561 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.886111 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.887272 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.887589 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.887704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:41.890165 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:41.890168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:41.890171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:41.890199 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:41.891534 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:41.892131 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:41.892134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:41.892135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:41.892228 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:41.892624 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:41.892627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:41.892629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:41.892886 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:41.892972 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:41.893691 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 00:53:41.893814 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:53:41.895083 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:41.882132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45273 (local address 127.16.202.2:56813)
0925 00:53:41.882612 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:41.882657 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:41.883872 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:41.884452 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:41.884460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:41.884878 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:41.885397 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:41.885411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.886223 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.886226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:41.887128 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:41.887136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:41.887734 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:41.887737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:41.887897 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:41.888456 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:41.888472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:41.890004 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:41.891643 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:41.891650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:41.891668 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:41.892012 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:41.892321 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:41.892324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:41.892326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:41.892503 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:41.892982 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:41.892989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:41.893334 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 00:53:41.893561 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 00:53:41.893935 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260925 00:53:41.896384 23092 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45273
I20260925 00:53:41.896628 23092 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:41.897159 23092 heartbeater.cc:511] Master 127.16.202.62:45273 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:41.898324 22767 ts_manager.cc:194] Registered new tserver with Master: 86e66345e8fa41ba96806fe6a71b8503 (127.16.202.2:34053)
I20260925 00:53:41.898927 22767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:56813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:41.918262 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45273
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:42.028757 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:42.029028 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:42.029074 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:42.033098 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:42.033164 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:42.033231 23097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:42.038547 23097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:42.039651 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:42.040719 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:42.050199 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:42.050271 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:42.050271 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:42.050469 23097 server_base.cc:1034] running on GCE node
I20260925 00:53:42.050951 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:42.051563 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:42.052742 23097 hybrid_clock.cc:648] HybridClock initialized: now 1790297622052722 us; error 32 us; skew 500 ppm
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.055506 23097 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:42.056433 23097 webserver.cc:460] Webserver started at http://127.16.202.3:45863/ using document root <none> and password file <none>
I20260925 00:53:42.057013 23097 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:42.057075 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:42.057250 23097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:42.059324 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "8e1f14762ccb4dc486c3ebf14de7a2cd"
format_stamp: "Formatted at 2026-09-25 00:53:42 on dist-test-slave-rpk1"
server_key: "78e4dc59782c3c90c673df91aaa3b0e9"
server_key_iv: "a3d751f0010ccba3a04a7a86fcb340aa"
server_key_version: "encryptionkey@0"
I20260925 00:53:42.059835 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "8e1f14762ccb4dc486c3ebf14de7a2cd"
format_stamp: "Formatted at 2026-09-25 00:53:42 on dist-test-slave-rpk1"
server_key: "78e4dc59782c3c90c673df91aaa3b0e9"
server_key_iv: "a3d751f0010ccba3a04a7a86fcb340aa"
server_key_version: "encryptionkey@0"
I20260925 00:53:42.063627 23097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 00:53:42.066222 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:42.067574 23097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:53:42.067687 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "8e1f14762ccb4dc486c3ebf14de7a2cd"
format_stamp: "Formatted at 2026-09-25 00:53:42 on dist-test-slave-rpk1"
server_key: "78e4dc59782c3c90c673df91aaa3b0e9"
server_key_iv: "a3d751f0010ccba3a04a7a86fcb340aa"
server_key_version: "encryptionkey@0"
I20260925 00:53:42.067791 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:42.089522 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:42.092617 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:42.092816 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:42.093400 23097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:42.094234 23097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:42.094283 23097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:42.094337 23097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:42.094369 23097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:42.103060 23097 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:33053
I20260925 00:53:42.103101 23225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:33053 every 8 connection(s)
I20260925 00:53:42.103957 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:42.105122 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23097
I20260925 00:53:42.105228 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:42.105535 17192 external_mini_cluster.cc:1428] Setting key 52cef673520616baec59f5bb80899ac3
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:42.117220 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:42.107027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:57267 (local address 127.16.202.62:45273)
0925 00:53:42.107143 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.107147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.107160 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:42.108139 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.108143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.108198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.108267 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.109529 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.110069 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.111171 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.111401 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.111487 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:42.113854 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:42.113858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:42.113860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:42.113886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:42.115215 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.115700 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.115703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.115704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.115781 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.116098 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.116101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.116103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.116291 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:42.116376 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:42.116932 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 00:53:42.117056 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:53:42.118091 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:42.106194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45273 (local address 127.16.202.3:57267)
0925 00:53:42.106658 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:42.106701 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:42.107926 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:42.108472 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:42.108479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:42.108884 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:42.109389 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:42.109402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.110185 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.110188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:42.111051 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:42.111060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.111537 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.111541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:42.111702 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:42.112237 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:42.112253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:42.113678 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:42.115325 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:42.115332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:42.115359 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:42.115606 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:42.115860 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:42.115863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:42.115865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:42.116014 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:42.116394 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:42.116402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:42.116659 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 00:53:42.116880 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 00:53:42.117230 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260925 00:53:42.119385 23226 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45273
I20260925 00:53:42.119637 23226 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:42.120167 23226 heartbeater.cc:511] Master 127.16.202.62:45273 requested a full tablet report, sending...
I20260925 00:53:42.121140 22767 ts_manager.cc:194] Registered new tserver with Master: 8e1f14762ccb4dc486c3ebf14de7a2cd (127.16.202.3:33053)
I20260925 00:53:42.121666 22767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:57267
I20260925 00:53:42.132258 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/test-admin.keytab.
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.150101 17192 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 00:53:42.153264 23233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:42.151716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.16.202.1:39245)
0925 00:53:42.152035 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.152051 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.152074 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:53:42.152124 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.152129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.152330 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.152449 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.153076 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53810: BlockingRecv error: recv got EOF from 127.0.0.1:53810 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":128,"threads_started":1}
W20260925 00:53:42.153554 17192 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc)
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.154597 17192 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.155565 17192 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:42.165858 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:42.156545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.16.202.1:39245)
0925 00:53:42.156706 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.156711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.156728 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:53:42.156850 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.156852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.156909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.156975 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.157584 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.158485 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.159397 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.159809 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.159893 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:42.162341 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:42.162351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:42.162364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:42.162389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:42.164028 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.164493 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.164498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.164500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.164574 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.164831 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.164833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.164834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.165085 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:42.165229 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:42.165551 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 00:53:42.165725 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 00:53:42.169716 23233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:42.168849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.16.202.1:39245)
0925 00:53:42.168971 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.168975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.169044 (+ 69us) server_negotiation.cc:407] Connection header received
0925 00:53:42.169156 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.169158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.169193 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.169248 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.169614 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53826: BlockingRecv error: recv got EOF from 127.0.0.1:53826 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 00:53:42.169790 17192 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc' of type FILE
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.170833 17192 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.171705 17192 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:42.180014 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:42.172529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.16.202.1:39245)
0925 00:53:42.172635 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.172639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.172741 (+ 102us) server_negotiation.cc:407] Connection header received
0925 00:53:42.172849 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.172851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.172886 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.172945 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.173397 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.173909 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.174686 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.174936 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.175023 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:42.177093 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:42.177096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:42.177098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:42.177119 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:42.178444 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.178820 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.178823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.178824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.178905 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.179208 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.179211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.179212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.179442 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:42.179542 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:42.179754 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 00:53:42.179851 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 00:53:42.182847 23233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:42.181888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.16.202.1:39245)
0925 00:53:42.182009 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.182013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.182131 (+ 118us) server_negotiation.cc:407] Connection header received
0925 00:53:42.182276 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.182278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.182314 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.182372 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.182759 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53832: BlockingRecv error: recv got EOF from 127.0.0.1:53832 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 00:53:42.183070 17192 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297591180183-17192-0/krb5kdc/krb5cc' of type FILE
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.184033 17192 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:42.184886 17192 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:53:42.193173 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:42.185693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.16.202.1:39245)
0925 00:53:42.185818 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:42.185821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:42.185867 (+ 46us) server_negotiation.cc:407] Connection header received
0925 00:53:42.185971 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:42.185973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:42.186011 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:42.186068 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:42.186536 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.187015 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.187810 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:42.188025 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:42.188100 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:42.190272 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:42.190275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:42.190277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:42.190337 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:42.191590 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.191997 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.192000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.192001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.192074 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:42.192380 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:42.192382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:42.192384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:42.192555 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:42.192648 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:42.192926 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 00:53:42.193018 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:42.194114 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22829
I20260925 00:53:42.198994 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22963
I20260925 00:53:42.203876 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23097
I20260925 00:53:42.208117 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 22735
2026-09-25T00:53:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2080 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[23246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[23246](info): set up 2 sockets
Sep 25 00:53:42 dist-test-slave-rpk1 krb5kdc[23246](info): commencing operation
krb5kdc: starting...
W20260925 00:53:43.161329 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:43 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab.
Entry for principal oryx/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab.
I20260925 00:53:43.284559 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45099
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab
--principal=oryx/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:43.395087 23259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:43.395396 23259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:43.395454 23259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:43.399516 23259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:43.399585 23259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:43.399611 23259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:43.399634 23259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:43.399663 23259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:43.405026 23259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:45099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab
--principal=oryx/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:43.406092 23259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:43.407095 23259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:43.415323 23264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:43.415403 23259 server_base.cc:1034] running on GCE node
W20260925 00:53:43.415338 23265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:43.415346 23267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:43.416137 23259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:43.417287 23259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:43.418483 23259 hybrid_clock.cc:648] HybridClock initialized: now 1790297623418463 us; error 31 us; skew 500 ppm
Sep 25 00:53:43 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:43.421358 23259 init.cc:377] Logged in from keytab as oryx/127.16.202.62@KRBTEST.COM (short username oryx)
I20260925 00:53:43.422330 23259 webserver.cc:460] Webserver started at http://127.16.202.62:45785/ using document root <none> and password file <none>
I20260925 00:53:43.422863 23259 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:43.422924 23259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:43.423105 23259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:43.425274 23259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "34231b5a95474f3ebf353786e5000770"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "59420bd72be8cb631d0b7a1bc1e7af88"
server_key_iv: "2c7ecb18d886af56624977bb2dcd7a88"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.425791 23259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "34231b5a95474f3ebf353786e5000770"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "59420bd72be8cb631d0b7a1bc1e7af88"
server_key_iv: "2c7ecb18d886af56624977bb2dcd7a88"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.429554 23259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:53:43.432111 23274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.433519 23259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:53:43.433635 23259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "34231b5a95474f3ebf353786e5000770"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "59420bd72be8cb631d0b7a1bc1e7af88"
server_key_iv: "2c7ecb18d886af56624977bb2dcd7a88"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.433743 23259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:43.463068 23259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:43.466461 23259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:43.466639 23259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:43.473620 23259 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:45099
I20260925 00:53:43.473910 23326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:45099 every 8 connection(s)
I20260925 00:53:43.474669 23259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:43.477746 23327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:43.481299 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23259
I20260925 00:53:43.481463 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:43.481752 17192 external_mini_cluster.cc:1428] Setting key 736821fd01c2e14937215031ebcd85a2
I20260925 00:53:43.484375 23327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Bootstrap starting.
I20260925 00:53:43.486665 23327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:43.487409 23327 log.cc:824] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:43.489354 23327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: No bootstrap required, opened a new log
Sep 25 00:53:43 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:43.492661 23327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } }
I20260925 00:53:43.492947 23327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:43.492988 23327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34231b5a95474f3ebf353786e5000770, State: Initialized, Role: FOLLOWER
I20260925 00:53:43.493448 23327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } }
I20260925 00:53:43.493572 23327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:43.493635 23327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:43.493731 23327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:43.494630 23327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } }
I20260925 00:53:43.494987 23327 leader_election.cc:302] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34231b5a95474f3ebf353786e5000770; no voters:
I20260925 00:53:43.495298 23327 leader_election.cc:288] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:43.495409 23332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:43.495613 23332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 LEADER]: Becoming Leader. State: Replica: 34231b5a95474f3ebf353786e5000770, State: Running, Role: LEADER
I20260925 00:53:43.495977 23332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } }
I20260925 00:53:43.496624 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:43.483244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.16.202.62:45099)
0925 00:53:43.483779 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:43.483795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:43.483833 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:53:43.484476 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:43.484501 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:43.484921 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:43.485260 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:43.486075 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:43.487116 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:43.487951 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:43.488451 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:43.488596 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:43.491256 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:43.491262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:43.491278 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:43.491307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:43.493482 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:43.494004 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:43.494009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:43.494014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:43.494130 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:43.494399 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:43.494403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:43.494404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:43.494853 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:43.495033 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:43.495354 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 00:53:43.495795 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":174,"threads_started":1}
I20260925 00:53:43.496738 23327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:43.498803 23333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } }
I20260925 00:53:43.498958 23333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:43.499627 23334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34231b5a95474f3ebf353786e5000770. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } }
I20260925 00:53:43.499768 23334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
W20260925 00:53:43.501753 23346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:53:43.501827 23346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:53:43.501916 23347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:43.502619 23347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:43.508716 23347 catalog_manager.cc:1347] Generated new cluster ID: cc3ac59429fa4eada86d966bc6f45633
I20260925 00:53:43.508808 23347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:43.537696 23347 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:43.538794 23347 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:43.548784 23347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Generated new TSK 0
I20260925 00:53:43.549701 23347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab.
Entry for principal oryx/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab.
I20260925 00:53:43.580006 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45099
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:43.697556 23353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:43.697883 23353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:43.697929 23353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:43.702399 23353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:43.702490 23353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:43.702553 23353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:43.708313 23353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:43.709542 23353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:43.710650 23353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:43.723212 23353 server_base.cc:1034] running on GCE node
W20260925 00:53:43.723104 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:43.723157 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:43.723114 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:43.723753 23353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:43.724463 23353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:43.725636 23353 hybrid_clock.cc:648] HybridClock initialized: now 1790297623725612 us; error 67 us; skew 500 ppm
Sep 25 00:53:43 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:43.728475 23353 init.cc:377] Logged in from keytab as oryx/127.16.202.1@KRBTEST.COM (short username oryx)
I20260925 00:53:43.729450 23353 webserver.cc:460] Webserver started at http://127.16.202.1:46163/ using document root <none> and password file <none>
I20260925 00:53:43.730008 23353 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:43.730077 23353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:43.730269 23353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:43.732602 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "62c33483c2e14ceb81cdb603d4dcbce4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "92512f0a42ecfd0701e7c304ebbcff05"
server_key_iv: "bc7cf0ac5a6b921a86132b6fae439a74"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.733250 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "62c33483c2e14ceb81cdb603d4dcbce4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "92512f0a42ecfd0701e7c304ebbcff05"
server_key_iv: "bc7cf0ac5a6b921a86132b6fae439a74"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.737268 23353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:43.739995 23368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.741473 23353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:43.741611 23353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "62c33483c2e14ceb81cdb603d4dcbce4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "92512f0a42ecfd0701e7c304ebbcff05"
server_key_iv: "bc7cf0ac5a6b921a86132b6fae439a74"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.741729 23353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:43.764870 23353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:43.768508 23353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:43.768735 23353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:43.769363 23353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:43.770221 23353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:43.770269 23353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.770323 23353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:43.770357 23353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.779134 23353 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43423
I20260925 00:53:43.779167 23481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43423 every 8 connection(s)
I20260925 00:53:43.780066 23353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:43.787079 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23353
I20260925 00:53:43.787204 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:43.787509 17192 external_mini_cluster.cc:1428] Setting key b87b052068c6d72d2bcde92ec196d52f
Sep 25 00:53:43 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:43.794194 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:43.781939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:60959 (local address 127.16.202.62:45099)
0925 00:53:43.782067 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:43.782071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:43.782929 (+ 858us) server_negotiation.cc:407] Connection header received
0925 00:53:43.784224 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:43.784227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:43.784275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:43.784396 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:43.785630 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:43.786160 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:43.787381 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:43.787710 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:43.787835 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:43.790553 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:43.790558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:43.790568 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:43.790597 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:43.791929 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:43.792530 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:43.792533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:43.792534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:43.792618 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:43.793028 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:43.793031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:43.793032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:43.793222 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:43.793299 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:43.793948 (+ 649us) server_negotiation.cc:299] Negotiation successful
0925 00:53:43.794061 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:43.795416 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:43.782318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:60959)
0925 00:53:43.782772 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:43.782817 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:43.784026 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:43.784583 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:43.784590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:43.784988 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:43.785493 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:43.785508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:43.786275 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:43.786278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:43.787211 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:43.787222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:43.787868 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:43.787872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:43.788031 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:43.788657 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:43.788701 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:43.790376 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:43.792050 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:43.792057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:43.792074 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:43.792415 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:43.792729 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:43.792732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:43.792735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:43.792937 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:43.793323 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:43.793331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:43.793684 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 00:53:43.793925 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 00:53:43.794299 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260925 00:53:43.796721 23482 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45099
WARNING: no policy specified for oryx/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:43.796952 23482 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:43.797482 23482 heartbeater.cc:511] Master 127.16.202.62:45099 requested a full tablet report, sending...
I20260925 00:53:43.799086 23287 ts_manager.cc:194] Registered new tserver with Master: 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:43.800284 23287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:60959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab.
Entry for principal oryx/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab.
I20260925 00:53:43.815963 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45099
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:43.928216 23487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:43.928480 23487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:43.928524 23487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:43.932586 23487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:43.932657 23487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:43.932754 23487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:43.938163 23487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:43.939287 23487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:43.940313 23487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:43.950460 23495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:43.950435 23493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:43.950618 23487 server_base.cc:1034] running on GCE node
W20260925 00:53:43.950394 23492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:43.951236 23487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:43.951982 23487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:43.953143 23487 hybrid_clock.cc:648] HybridClock initialized: now 1790297623953095 us; error 64 us; skew 500 ppm
Sep 25 00:53:43 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:43.955997 23487 init.cc:377] Logged in from keytab as oryx/127.16.202.2@KRBTEST.COM (short username oryx)
I20260925 00:53:43.956990 23487 webserver.cc:460] Webserver started at http://127.16.202.2:37059/ using document root <none> and password file <none>
I20260925 00:53:43.957557 23487 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:43.957623 23487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:43.957828 23487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:43.959960 23487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "ee1795c71d5741c68314cbac9a9efea4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "25207c1b2d5bc16e10f5c48f36985666"
server_key_iv: "42d5df7532022b522f2a3589a016d898"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.960520 23487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "ee1795c71d5741c68314cbac9a9efea4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "25207c1b2d5bc16e10f5c48f36985666"
server_key_iv: "42d5df7532022b522f2a3589a016d898"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.964385 23487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 00:53:43.967207 23502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.968552 23487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:43.968662 23487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "ee1795c71d5741c68314cbac9a9efea4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "25207c1b2d5bc16e10f5c48f36985666"
server_key_iv: "42d5df7532022b522f2a3589a016d898"
server_key_version: "encryptionkey@0"
I20260925 00:53:43.968819 23487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:43.983480 23487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:43.987097 23487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:43.987291 23487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:43.987943 23487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:43.988838 23487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:43.988888 23487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.988942 23487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:43.988972 23487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:43.997560 23487 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:39921
I20260925 00:53:43.997618 23615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:39921 every 8 connection(s)
I20260925 00:53:43.998525 23487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:44.002557 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23487
I20260925 00:53:44.002657 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:44.002912 17192 external_mini_cluster.cc:1428] Setting key 0f0a56310771eb443adfeea51cb27c4c
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:53:44.013262 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.000623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:39195 (local address 127.16.202.62:45099)
0925 00:53:44.000789 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.000793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.001767 (+ 974us) server_negotiation.cc:407] Connection header received
0925 00:53:44.003167 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.003170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.003228 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.003310 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.004835 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.005394 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.006559 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.006803 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.006907 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.009529 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.009533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.009536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.009570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.011146 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.011650 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.011654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.011656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.011746 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.012086 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.012089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.012091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.012299 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.012389 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.013021 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.013136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 00:53:44.014190 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.001070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:39195)
0925 00:53:44.001595 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.001643 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.002944 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.003517 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.003525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.004016 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.004632 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.004647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.005532 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.005534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.006428 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.006447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.007047 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.007051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.007227 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.007817 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.007834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.009338 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.011260 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.011266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.011280 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.011540 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.011824 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.011826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.011827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.011985 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.012412 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.012420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.012714 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.012953 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.013241 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:44.015564 23616 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45099
I20260925 00:53:44.015794 23616 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:44.016350 23616 heartbeater.cc:511] Master 127.16.202.62:45099 requested a full tablet report, sending...
I20260925 00:53:44.017570 23287 ts_manager.cc:194] Registered new tserver with Master: ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921)
I20260925 00:53:44.018164 23287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:39195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab.
Entry for principal oryx/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab.
I20260925 00:53:44.031467 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45099
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:44.144142 23621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:44.144459 23621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:44.144503 23621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:44.149010 23621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:44.149078 23621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:44.149140 23621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:44.154583 23621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:44.155731 23621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:44.156831 23621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:44.166558 23629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.166564 23626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.166564 23627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.166849 23621 server_base.cc:1034] running on GCE node
I20260925 00:53:44.167402 23621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:44.168054 23621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:44.169261 23621 hybrid_clock.cc:648] HybridClock initialized: now 1790297624169211 us; error 58 us; skew 500 ppm
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:44.172492 23621 init.cc:377] Logged in from keytab as oryx/127.16.202.3@KRBTEST.COM (short username oryx)
I20260925 00:53:44.173638 23621 webserver.cc:460] Webserver started at http://127.16.202.3:40565/ using document root <none> and password file <none>
I20260925 00:53:44.174196 23621 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.174273 23621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.174458 23621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:44.176580 23621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "0719fa59798645649e691f3a6c185032"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "32c56a1d549a5c3f7d6328e722f9a71d"
server_key_iv: "e817f96c846e821f11d0f7288dfe1f07"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.177255 23621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "0719fa59798645649e691f3a6c185032"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "32c56a1d549a5c3f7d6328e722f9a71d"
server_key_iv: "e817f96c846e821f11d0f7288dfe1f07"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.181195 23621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:53:44.183840 23636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.185227 23621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:44.185374 23621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "0719fa59798645649e691f3a6c185032"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "32c56a1d549a5c3f7d6328e722f9a71d"
server_key_iv: "e817f96c846e821f11d0f7288dfe1f07"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.185465 23621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.202390 23621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:44.205745 23621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:44.205906 23621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:44.206459 23621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:44.207263 23621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:44.207311 23621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.207350 23621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:44.207365 23621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.215945 23621 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:44973
I20260925 00:53:44.216025 23749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:44973 every 8 connection(s)
I20260925 00:53:44.216951 23621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:44.217918 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23621
I20260925 00:53:44.218016 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:44.218318 17192 external_mini_cluster.cc:1428] Setting key 18ef40377eb07615574902cd08d38d37
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:44.232283 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.219066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:40729 (local address 127.16.202.62:45099)
0925 00:53:44.219210 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.219214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.220847 (+ 1633us) server_negotiation.cc:407] Connection header received
0925 00:53:44.222194 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.222197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.222244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.222358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.223621 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.224168 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.225400 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.225628 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.225710 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.228467 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.228471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.228473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.228503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.229888 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.230474 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.230478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.230479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.230570 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.230946 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.230949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.230951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.231202 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.231315 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.231984 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.232115 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:53:44.233302 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.219570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:40729)
0925 00:53:44.220666 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.220749 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.221979 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.222527 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.222534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.222951 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.223476 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.223490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.224268 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.224270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.225272 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.225281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.225753 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.225756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.225924 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.226510 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.226528 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.228204 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.230028 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.230034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.230051 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.230349 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.230657 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.230659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.230661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.230832 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.231342 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.231351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.231674 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.231910 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.232273 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":129,"threads_started":1}
I20260925 00:53:44.234527 23750 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45099
I20260925 00:53:44.234812 23750 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:44.235347 23750 heartbeater.cc:511] Master 127.16.202.62:45099 requested a full tablet report, sending...
I20260925 00:53:44.236501 23287 ts_manager.cc:194] Registered new tserver with Master: 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973)
I20260925 00:53:44.237161 23287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:40729
I20260925 00:53:44.245051 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.202.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.61.keytab.
Entry for principal oryx/127.16.202.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.61.keytab.
I20260925 00:53:44.272210 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:42451
--webserver_interface=127.16.202.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.61.keytab
--principal=oryx/127.16.202.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:45099,127.16.202.61:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:44.382136 23755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:44.382432 23755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:44.382472 23755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:44.386524 23755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:44.386597 23755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:44.386615 23755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:44.386632 23755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:44.386646 23755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:44.392146 23755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:45099,127.16.202.61:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.61.keytab
--principal=oryx/127.16.202.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.202.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
W20260925 00:53:44.392383 23755 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.202.62:45099,127.16.202.61:42451'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 00:53:44.393303 23755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:44.394287 23755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:44.402762 23760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.402760 23761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.402858 23755 server_base.cc:1034] running on GCE node
W20260925 00:53:44.402755 23763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.403612 23755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:44.404785 23755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:44.405936 23755 hybrid_clock.cc:648] HybridClock initialized: now 1790297624405915 us; error 47 us; skew 500 ppm
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:44.408565 23755 init.cc:377] Logged in from keytab as oryx/127.16.202.61@KRBTEST.COM (short username oryx)
I20260925 00:53:44.409479 23755 webserver.cc:460] Webserver started at http://127.16.202.61:39993/ using document root <none> and password file <none>
I20260925 00:53:44.410007 23755 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.410070 23755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.410254 23755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:44.412362 23755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/instance:
uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "d60ffebc22429ea15bc992bca0d8ce96"
server_key_iv: "e781fba7d3f40ced451e22594a6401c2"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.412947 23755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal/instance:
uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "d60ffebc22429ea15bc992bca0d8ce96"
server_key_iv: "e781fba7d3f40ced451e22594a6401c2"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.416844 23755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:53:44.419420 23770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.420845 23755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:44.420974 23755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "d60ffebc22429ea15bc992bca0d8ce96"
server_key_iv: "e781fba7d3f40ced451e22594a6401c2"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.421085 23755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.439536 23755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:44.443234 23755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:44.443457 23755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:44.450399 23755 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.61:42451
I20260925 00:53:44.450680 23822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.61:42451 every 8 connection(s)
I20260925 00:53:44.451409 23755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
I20260925 00:53:44.454463 23823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:44.458348 23823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:53:44.458765 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23755
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.61@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
I20260925 00:53:44.471256 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.460915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.16.202.62:45099)
0925 00:53:44.461043 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.461047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.461331 (+ 284us) server_negotiation.cc:407] Connection header received
0925 00:53:44.461739 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.461741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.461787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.461875 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.462709 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.463242 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.464241 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.464462 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.464552 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.467323 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.467326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.467328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.467355 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.468816 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.469460 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.469464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.469465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.469566 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.469962 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.469965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.469968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.470276 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.470384 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.470945 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.471087 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:53:44.472321 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.460187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.16.202.61:42451)
0925 00:53:44.460586 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.460600 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.460634 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:53:44.461486 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.461509 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.461748 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.461913 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.462722 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.463576 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.464369 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.464796 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.464938 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.467712 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.467720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.467735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.467760 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.469483 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.469994 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.470000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.470002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.470069 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.470356 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.470359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.470361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.470729 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.470904 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.471121 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.471565 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260925 00:53:44.472321 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.460839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:46648)
0925 00:53:44.461177 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.461230 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.461553 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.461944 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.461950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.462305 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.462560 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.462573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.463362 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.463365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.464125 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.464133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.464599 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.464603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.464820 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.465413 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.465434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.467166 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.468946 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.468953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.468965 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.469296 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.469711 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.469714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.469716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.469869 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.470369 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.470376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.470717 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.470893 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.471263 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260925 00:53:44.474553 23823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.61@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
I20260925 00:53:44.485013 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.475112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46234)
0925 00:53:44.475208 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.475222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.475292 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.476969 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.476972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.477162 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.477287 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.477294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.477987 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.477990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.478718 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.478728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.479828 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.479838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.479943 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.480436 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.480449 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.481981 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.483555 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.483559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.483561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.483792 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.484036 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.484039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.484042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.484123 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.484425 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.484428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.484518 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.484606 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.484789 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 00:53:44.485013 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.475187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.16.202.61:42451)
0925 00:53:44.476788 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.476791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.476801 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:53:44.476833 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.476835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.476870 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.476927 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.477384 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.477897 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.478864 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.479680 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.479777 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.482134 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.482137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.482139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.482161 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.483461 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.483880 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.483882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.483900 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.483967 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.484190 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.484192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.484193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.484356 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.484424 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.484653 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.484844 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":1534}
I20260925 00:53:44.488189 23823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Bootstrap starting.
I20260925 00:53:44.490633 23823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:44.491379 23823 log.cc:824] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:44.493381 23823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: No bootstrap required, opened a new log
I20260925 00:53:44.496310 23823 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } }
I20260925 00:53:44.496517 23823 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:44.496580 23823 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbde26b7bf7e4a4b9d1eeeeaa458ea34, State: Initialized, Role: FOLLOWER
I20260925 00:53:44.497088 23823 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } }
I20260925 00:53:44.497591 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } } }
I20260925 00:53:44.497705 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:44.498328 23823 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 00:53:44.502856 23840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:53:44.502962 23840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:53:44.505100 23755 master_runner.cc:428] Detected that this master bbde26b7bf7e4a4b9d1eeeeaa458ea34 is joining an existing cluster
I20260925 00:53:44.505154 23755 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.202.61:42451
I20260925 00:53:44.515384 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.507225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.16.202.62:45099)
0925 00:53:44.507347 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.507351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.507951 (+ 600us) server_negotiation.cc:407] Connection header received
0925 00:53:44.508073 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.508075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.508118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.508193 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.508716 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.509355 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.510545 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.510884 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.510968 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.512086 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.512089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.512092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.512117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.513748 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514220 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.514224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.514226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.514306 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514548 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.514551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.514553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.514775 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.514879 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.515117 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.515231 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:53:44.515408 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.507145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:46658)
0925 00:53:44.507864 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.507878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.507982 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.508220 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.508223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.508467 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.508598 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.508605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.509470 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.509473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.510443 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.510453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.511066 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.511068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.511131 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.511617 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.511628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.511954 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.513848 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.513852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.513854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.514133 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.514392 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.514397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.514471 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.514872 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.514875 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.514969 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.515054 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.515201 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":30,"thread_start_us":101,"threads_started":1}
I20260925 00:53:44.515849 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.507645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.16.202.61:42451)
0925 00:53:44.507745 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.507750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.508274 (+ 524us) server_negotiation.cc:407] Connection header received
0925 00:53:44.508463 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.508465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.508505 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.508581 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.509110 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.509700 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.510494 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.510797 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.510898 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.512536 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.512540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.512542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.512561 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.514119 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514550 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.514553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.514555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.514644 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514892 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.514895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.514897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.515073 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.515131 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.515406 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.515706 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:53:44.516490 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.507560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46242)
0925 00:53:44.508194 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.508206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.508320 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.508614 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.508616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.508875 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.509001 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.509008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.509791 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.509794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.510399 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.510407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.510913 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.510917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.511008 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.511954 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.511968 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.512380 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.514225 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.514230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.514458 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.514731 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.514733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.514735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.514809 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.515183 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.515185 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.515265 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.516180 (+ 915us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.516325 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":37,"thread_start_us":95,"threads_started":1}
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.62@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
I20260925 00:53:44.530346 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.520595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.16.202.61:42451)
0925 00:53:44.520754 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.520759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.521150 (+ 391us) server_negotiation.cc:407] Connection header received
0925 00:53:44.521351 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.521353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.521406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.521470 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.522325 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.522832 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.524145 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.524386 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.524477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.527142 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.527146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.527148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.527173 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.528532 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.529062 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.529066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.529068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.529147 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.529421 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.529425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.529427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.529614 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.529695 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.530062 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.530183 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:53:44.530408 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.520570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46244)
0925 00:53:44.521004 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.521069 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.521219 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.521526 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.521532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.522033 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.522174 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.522182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.522954 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.522958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.524013 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.524025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.524501 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.524505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.524622 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.525335 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.525351 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.526984 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.528649 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.528663 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.528698 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.528967 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.529231 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.529234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.529236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.529331 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.529714 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.529723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.529853 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.529982 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.530241 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":119,"threads_started":1}
I20260925 00:53:44.532164 23287 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.202.61:42451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34"
member_type: NON_VOTER
last_known_addr {
host: "127.16.202.61"
port: 42451
}
attrs {
promote: true
}
}
dest_uuid: "34231b5a95474f3ebf353786e5000770"
cas_config_opid_index: -1
I20260925 00:53:44.532907 23287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } }
I20260925 00:53:44.534835 23851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bbde26b7bf7e4a4b9d1eeeeaa458ea34 (127.16.202.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.16.202.60@KRBTEST.COM; defaulting to no policy
I20260925 00:53:44.535552 23797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:44.536643 23851 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.202.61:42451
I20260925 00:53:44.536903 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } }
I20260925 00:53:44.537003 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:44.537145 23755 master.cc:537] Master@127.16.202.61:42451 shutting down...
I20260925 00:53:44.537776 23797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Refusing update from remote peer 34231b5a95474f3ebf353786e5000770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.60@KRBTEST.COM" created.
I20260925 00:53:44.538558 23850 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:53:44.540787 23278 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 00:53:44.545223 23755 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:44.545545 23755 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:44.545657 23755 tablet_replica.cc:330] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: stopping tablet replica
W20260925 00:53:44.546046 23278 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 -> Peer bbde26b7bf7e4a4b9d1eeeeaa458ea34 (127.16.202.61:42451): Couldn't send request to peer bbde26b7bf7e4a4b9d1eeeeaa458ea34. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.60.keytab.
Entry for principal oryx/127.16.202.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.60.keytab.
I20260925 00:53:44.556340 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:37597
--webserver_interface=127.16.202.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.60.keytab
--principal=oryx/127.16.202.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:53:44.560166 23755 master.cc:552] Master@127.16.202.61:42451 shutdown complete.
I20260925 00:53:44.560241 23755 master_runner.cc:305] Clearing existing system tablet
I20260925 00:53:44.560468 23755 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.560518 23755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.561669 23755 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:44.562453 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.562688 23755 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.562821 23755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:44.562870 23755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "d60ffebc22429ea15bc992bca0d8ce96"
server_key_iv: "e781fba7d3f40ced451e22594a6401c2"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.564641 23755 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:44.566744 23755 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 00:53:44.566838 23755 log.cc:1196] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 00:53:44.567085 23755 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Deleting consensus metadata
I20260925 00:53:44.567245 23755 master_runner.cc:315] Copying system tablet from 127.16.202.62:45099
I20260925 00:53:44.568325 23755 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: tablet copy: Beginning tablet copy session from remote peer at address 127.16.202.62:45099
I20260925 00:53:44.575848 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.568907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:46666)
0925 00:53:44.569208 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.569221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.569337 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.569599 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.569602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.569795 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.569893 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.569899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.570707 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.570711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.571285 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.571294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.571772 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.571776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.571869 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.572365 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.572377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.572741 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.574424 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.574428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.574430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.574630 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.574882 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.574885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.574887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.574966 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.575317 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.575320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.575420 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.575504 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.575650 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":82,"threads_started":1}
I20260925 00:53:44.575837 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.569015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.16.202.62:45099)
0925 00:53:44.569146 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.569150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.569299 (+ 149us) server_negotiation.cc:407] Connection header received
0925 00:53:44.569455 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.569458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.569508 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.569569 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.570017 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.570600 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.571402 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.571625 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.571714 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.572938 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.572942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.572945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.572973 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.574322 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.574718 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.574721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.574722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.574797 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.575042 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.575045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.575047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.575231 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.575317 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.575568 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.575676 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:53:44.576772 23311 tablet_copy_service.cc:140] P 34231b5a95474f3ebf353786e5000770: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bbde26b7bf7e4a4b9d1eeeeaa458ea34 ({username='oryx', principal='oryx/127.16.202.61@KRBTEST.COM'} at 127.0.0.1:46666)
I20260925 00:53:44.576990 23311 tablet_copy_service.cc:161] P 34231b5a95474f3ebf353786e5000770: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bbde26b7bf7e4a4b9d1eeeeaa458ea34 at {username='oryx', principal='oryx/127.16.202.61@KRBTEST.COM'} at 127.0.0.1:46666: session id = bbde26b7bf7e4a4b9d1eeeeaa458ea34-00000000000000000000000000000000
I20260925 00:53:44.578884 23311 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 00:53:44.580598 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 00:53:44.582544 23755 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: tablet copy: Starting download of 0 data blocks...
I20260925 00:53:44.582716 23755 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: tablet copy: Starting download of 1 WAL segments...
I20260925 00:53:44.584311 23755 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 00:53:44.585919 23311 tablet_copy_service.cc:342] P 34231b5a95474f3ebf353786e5000770: Request end of tablet copy session bbde26b7bf7e4a4b9d1eeeeaa458ea34-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.202.61@KRBTEST.COM'} at 127.0.0.1:46666
I20260925 00:53:44.586032 23311 tablet_copy_service.cc:433] P 34231b5a95474f3ebf353786e5000770: ending tablet copy session bbde26b7bf7e4a4b9d1eeeeaa458ea34-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bbde26b7bf7e4a4b9d1eeeeaa458ea34
I20260925 00:53:44.593420 23755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:44.595155 23864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.595155 23867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.595420 23755 server_base.cc:1034] running on GCE node
W20260925 00:53:44.595252 23865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.595824 23755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:44.596107 23755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:44.597229 23755 hybrid_clock.cc:648] HybridClock initialized: now 1790297624597220 us; error 25 us; skew 500 ppm
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:44.598613 23755 init.cc:377] Logged in from keytab as oryx/127.16.202.61@KRBTEST.COM (short username oryx)
I20260925 00:53:44.599035 23755 webserver.cc:460] Webserver started at http://127.16.202.61:35259/ using document root <none> and password file <none>
I20260925 00:53:44.599207 23755 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.599241 23755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.605162 23755 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 00:53:44.606278 23874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.606499 23755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:44.606565 23755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "d60ffebc22429ea15bc992bca0d8ce96"
server_key_iv: "e781fba7d3f40ced451e22594a6401c2"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.606653 23755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.615123 23755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:44.616931 23755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:44.621294 23755 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.61:42451
I20260925 00:53:44.621903 23755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
I20260925 00:53:44.623273 23927 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:53:44.624147 23927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Bootstrap starting.
I20260925 00:53:44.628834 23926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.61:42451 every 8 connection(s)
I20260925 00:53:44.632838 23927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:53:44.633145 23927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Bootstrap complete.
I20260925 00:53:44.633715 23927 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } }
I20260925 00:53:44.633838 23927 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bbde26b7bf7e4a4b9d1eeeeaa458ea34, State: Initialized, Role: LEARNER
I20260925 00:53:44.633942 23927 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } }
I20260925 00:53:44.634294 23930 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } }
I20260925 00:53:44.634442 23927 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:44.634466 23930 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: LEARNER
I20260925 00:53:44.637821 23941 catalog_manager.cc:1259] Loaded cluster ID: cc3ac59429fa4eada86d966bc6f45633
I20260925 00:53:44.637881 23941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: loading cluster ID for follower catalog manager: success
I20260925 00:53:44.639596 23941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: acquiring CA information for follower catalog manager: success
I20260925 00:53:44.640843 23941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 00:53:44.711411 23855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:44.711719 23855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:44.711779 23855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:44.716307 23855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:44.716406 23855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:44.716434 23855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:44.716455 23855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:44.716477 23855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:44.722451 23855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.60.keytab
--principal=oryx/127.16.202.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.202.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:44.723568 23855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:44.724622 23855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:44.733013 23949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.733026 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.733088 23855 server_base.cc:1034] running on GCE node
W20260925 00:53:44.733105 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.733843 23855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:44.734983 23855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:44.736181 23855 hybrid_clock.cc:648] HybridClock initialized: now 1790297624736141 us; error 53 us; skew 500 ppm
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:44.751276 23855 init.cc:377] Logged in from keytab as oryx/127.16.202.60@KRBTEST.COM (short username oryx)
I20260925 00:53:44.752318 23855 webserver.cc:460] Webserver started at http://127.16.202.60:34007/ using document root <none> and password file <none>
I20260925 00:53:44.752987 23855 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.753067 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.753293 23855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:44.755450 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/instance:
uuid: "f0142bd70c61423082fa7eceb92b5fbc"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "7a715ceafc55751755249c57bb7f6b80"
server_key_iv: "0c74740319b9efdcda33ace8bbacee46"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.755983 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal/instance:
uuid: "f0142bd70c61423082fa7eceb92b5fbc"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "7a715ceafc55751755249c57bb7f6b80"
server_key_iv: "0c74740319b9efdcda33ace8bbacee46"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.760076 23855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:44.762781 23956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.764209 23855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:44.764314 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
uuid: "f0142bd70c61423082fa7eceb92b5fbc"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "7a715ceafc55751755249c57bb7f6b80"
server_key_iv: "0c74740319b9efdcda33ace8bbacee46"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.764407 23855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.783015 23855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:44.786051 23855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:44.786206 23855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:44.793233 23855 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.60:37597
I20260925 00:53:44.793486 24008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.60:37597 every 8 connection(s)
I20260925 00:53:44.794282 23855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
I20260925 00:53:44.797216 24009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:44.801590 24009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:53:44.802639 23482 heartbeater.cc:503] Master 127.16.202.62:45099 was elected leader, sending a full tablet report...
I20260925 00:53:44.804289 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 23855
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.202.60@KRBTEST.COM
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.60@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:44.815299 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.803572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.16.202.62:45099)
0925 00:53:44.803706 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.803710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.804624 (+ 914us) server_negotiation.cc:407] Connection header received
0925 00:53:44.805785 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.805789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.805845 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.805914 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.806931 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.807463 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.808532 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.808801 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.808890 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.812185 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.812188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.812190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.812215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.813594 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.814050 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.814054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.814056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.814129 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.814373 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.814377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.814379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.814563 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.814648 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.815027 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.815148 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:44.816184 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.805193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.16.202.60:37597)
0925 00:53:44.805535 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.805544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.805577 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:53:44.805711 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.805725 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.805913 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.806073 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.806901 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.807757 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.808599 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.809006 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.809114 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.811651 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.811657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.811669 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.811694 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.813322 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.813755 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.813761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.813762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.813888 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.814149 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.814153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.814155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.814591 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.814782 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.815000 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.815279 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":258,"spinlock_wait_cycles":2432,"thread_start_us":168,"threads_started":1}
I20260925 00:53:44.816190 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.803920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:46668)
0925 00:53:44.804462 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.804524 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.805592 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.805976 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.805986 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.806365 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.806782 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.806802 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.807578 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.807581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.808422 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.808432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.808949 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.808953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.809106 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.810100 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.810116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.812059 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.813695 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.813704 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.813715 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.813976 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.814211 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.814215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.814217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.814295 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.814662 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.814670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.814806 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.814970 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.815362 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":92,"spinlock_wait_cycles":4480,"thread_start_us":120,"threads_started":1}
I20260925 00:53:44.817461 24009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.60@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
I20260925 00:53:44.825708 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.817957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46258)
0925 00:53:44.818074 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.818088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.818152 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.818862 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.818865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.819014 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.819110 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.819115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.819979 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.819982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.820511 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.820518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.820996 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.820999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.821100 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.821594 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.821606 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.822817 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.824233 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.824237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.824239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.824455 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.824730 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.824733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.824735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.824809 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.825197 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.825200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.825291 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.825375 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.825523 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 00:53:44.825709 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.818424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.16.202.61:42451)
0925 00:53:44.818669 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.818673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.818684 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:53:44.818718 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.818720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.818758 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.818832 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.819218 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.819872 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.820617 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.820897 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.820988 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.822970 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.822972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.822974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.822999 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.824132 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.824546 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.824549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.824550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.824624 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.824905 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.824908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.824909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.825106 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.825194 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.825435 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.825553 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260925 00:53:44.826586 24009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.60@KRBTEST.COM for oryx/127.16.202.60@KRBTEST.COM
I20260925 00:53:44.833797 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.827091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53056)
0925 00:53:44.827202 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.827216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.827280 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.827497 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.827499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.827638 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.827712 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.827717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.828395 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.828398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.828947 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.828955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.829450 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.829453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.829509 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.829889 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.829901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.831010 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.832524 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.832526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.832528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.832761 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.833015 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.833017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.833019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.833096 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.833395 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.833397 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.833467 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.833533 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.833636 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 00:53:44.833822 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.827145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.16.202.60:37597)
0925 00:53:44.827253 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.827256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.827267 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:53:44.827380 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.827382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.827416 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.827477 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.827799 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.828283 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.829088 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.829349 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.829431 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.831145 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.831148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.831150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.831170 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.832428 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.832838 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.832841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.832842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.832916 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.833163 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.833165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.833166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.833331 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.833398 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.833573 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.833690 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260925 00:53:44.837129 24009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Bootstrap starting.
I20260925 00:53:44.839344 24009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:44.840034 24009 log.cc:824] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:44.841920 24009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: No bootstrap required, opened a new log
I20260925 00:53:44.844835 24009 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } }
I20260925 00:53:44.845058 24009 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:44.845098 24009 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0142bd70c61423082fa7eceb92b5fbc, State: Initialized, Role: FOLLOWER
I20260925 00:53:44.845480 24009 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } }
I20260925 00:53:44.845880 24015 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } } }
I20260925 00:53:44.845996 24015 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:44.846588 24009 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: configured and running, proceeding with master startup.
W20260925 00:53:44.850972 24026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:53:44.851049 24026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:53:44.854425 23855 master_runner.cc:428] Detected that this master f0142bd70c61423082fa7eceb92b5fbc is joining an existing cluster
I20260925 00:53:44.854493 23855 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.202.60:37597
I20260925 00:53:44.865387 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.858105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.16.202.60:37597)
0925 00:53:44.858183 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.858187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.858503 (+ 316us) server_negotiation.cc:407] Connection header received
0925 00:53:44.858620 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.858622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.858655 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.858695 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.859169 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.859896 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.861182 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.861504 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.861567 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.862659 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.862663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.862665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.862686 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.863876 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.864221 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.864234 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.864236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.864312 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.864545 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.864549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.864551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.864762 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.864845 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.865098 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.865256 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":25}
I20260925 00:53:44.865394 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.858193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53070)
0925 00:53:44.858441 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.858453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.858544 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.858816 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.858819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.858990 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.859081 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.859087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.860012 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.860015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.861073 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.861084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.861700 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.861703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.861798 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.862199 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.862212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.862545 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.863968 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.863971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.863974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.864143 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.864388 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.864391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.864393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.864466 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.864841 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.864844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.864940 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.865047 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.865190 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":22,"thread_start_us":76,"threads_started":1}
I20260925 00:53:44.869719 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23353
I20260925 00:53:44.872934 24015 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:44.873142 24015 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } }
I20260925 00:53:44.874109 24015 leader_election.cc:288] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099), bbde26b7bf7e4a4b9d1eeeeaa458ea34 (127.16.202.61:42451)
I20260925 00:53:44.874605 23901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0142bd70c61423082fa7eceb92b5fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" is_pre_election: true
I20260925 00:53:44.874606 23301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0142bd70c61423082fa7eceb92b5fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34231b5a95474f3ebf353786e5000770" is_pre_election: true
I20260925 00:53:44.874790 23901 raft_consensus.cc:1799] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Handling vote request from an unknown peer f0142bd70c61423082fa7eceb92b5fbc
I20260925 00:53:44.874928 23901 raft_consensus.cc:2396] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate f0142bd70c61423082fa7eceb92b5fbc for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 00:53:44.875602 23960 leader_election.cc:302] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0142bd70c61423082fa7eceb92b5fbc; no voters: 34231b5a95474f3ebf353786e5000770, bbde26b7bf7e4a4b9d1eeeeaa458ea34
I20260925 00:53:44.875872 24015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:44.876261 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:43423
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:45099
--webserver_port=46163
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:53:44.876988 24015 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 00:53:44.893541 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.856895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:46678)
0925 00:53:44.857157 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.857170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.857285 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.857717 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.857720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.857958 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.858115 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.858124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.859240 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.859243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.860189 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.860200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.860971 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.860974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.861105 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.861617 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.861630 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.889111 (+ 27481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.891826 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.891842 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.891845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.892077 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.892383 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.892389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.892489 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.892990 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.892993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.893091 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.893185 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.893339 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":28,"thread_start_us":82,"threads_started":1}
I20260925 00:53:44.893913 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.857512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.16.202.61:42451)
0925 00:53:44.857637 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.857640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.858069 (+ 429us) server_negotiation.cc:407] Connection header received
0925 00:53:44.858414 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.858417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.858474 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.858523 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.859115 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.859895 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.861424 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.861778 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.861879 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.890535 (+ 28656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.890540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.890543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.890576 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.892086 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892528 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.892532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.892534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.892618 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892929 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.892932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.892934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.893121 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.893203 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.893642 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.893770 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:44.894140 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.857012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.16.202.62:45099)
0925 00:53:44.857143 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.857148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.857248 (+ 100us) server_negotiation.cc:407] Connection header received
0925 00:53:44.857395 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.857399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.857463 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.857569 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.858263 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.859099 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.860328 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.860718 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.860799 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.889394 (+ 28595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.889398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.889401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.889440 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.891684 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892197 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.892200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.892202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.892280 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892581 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.892584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.892586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.892867 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.892951 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.893878 (+ 927us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.893990 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:53:44.894744 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.857436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46268)
0925 00:53:44.857990 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.858002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.858203 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.858651 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.858653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.858878 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.858991 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.858998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.860012 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.860015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.861308 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.861316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.861995 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.861999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.862087 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.889833 (+ 27746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.889850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.890328 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.892195 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.892200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.892421 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.892749 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.892752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.892754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.892831 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.893240 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.893242 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.893319 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.893412 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.893525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":3,"thread_start_us":371,"threads_started":1}
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297623, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.62@KRBTEST.COM for oryx/127.16.202.60@KRBTEST.COM
I20260925 00:53:44.905536 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.898171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53080)
0925 00:53:44.898360 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.898372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.898453 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.898715 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.898718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.898890 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.898996 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.899002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.899687 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.899691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.900232 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.900240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.900652 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.900656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.900847 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.901320 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.901334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.902641 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.904132 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.904136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.904138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.904330 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.904570 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.904573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.904575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.904651 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.905042 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.905045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.905134 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.905218 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.905364 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 00:53:44.905536 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.898222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.16.202.60:37597)
0925 00:53:44.898334 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.898339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.898413 (+ 74us) server_negotiation.cc:407] Connection header received
0925 00:53:44.898571 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.898574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.898623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.898685 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.899096 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.899593 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.900343 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.900559 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.900651 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.902770 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.902774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.902777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.902807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.904032 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.904412 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.904416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.904418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.904491 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.904768 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.904771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.904772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.904940 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.905046 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.905275 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.905385 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:53:44.906525 23287 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.202.60:37597: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc"
member_type: NON_VOTER
last_known_addr {
host: "127.16.202.60"
port: 37597
}
attrs {
promote: true
}
}
dest_uuid: "34231b5a95474f3ebf353786e5000770"
cas_config_opid_index: 5
I20260925 00:53:44.907137 23287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } }
I20260925 00:53:44.908375 23850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.909545 23850 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.202.60:37597
I20260925 00:53:44.909722 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.909829 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:44.910089 23855 master.cc:537] Master@127.16.202.60:37597 shutting down...
W20260925 00:53:44.912771 23278 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 -> Peer f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60:37597): Couldn't send request to peer f0142bd70c61423082fa7eceb92b5fbc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 00:53:44.914654 23855 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:44.914963 23855 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:44.915066 23855 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: stopping tablet replica
I20260925 00:53:44.918707 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.910106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46276)
0925 00:53:44.913054 (+ 2948us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.913068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.913287 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.913702 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.913705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.913721 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:44.913824 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.913830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.914832 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.914836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.917133 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.917148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.918230 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.918234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.918330 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.918394 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.918466 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.918559 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2874}
I20260925 00:53:44.919087 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.910499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.16.202.61:42451)
0925 00:53:44.912959 (+ 2460us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.912965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.913205 (+ 240us) server_negotiation.cc:407] Connection header received
0925 00:53:44.913417 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.913421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.913486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.913567 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:44.913951 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.914689 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.917272 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.917899 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.917995 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.918160 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.918892 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.918977 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2369}
I20260925 00:53:44.919535 23901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Refusing update from remote peer 34231b5a95474f3ebf353786e5000770: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 00:53:44.920032 24036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 00:53:44.922142 23901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.923991 23930 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34231b5a95474f3ebf353786e5000770. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.924098 23930 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: LEARNER
I20260925 00:53:44.924185 23930 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: NON_VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: true } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.924253 23930 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: LEARNER
I20260925 00:53:44.924996 23855 master.cc:552] Master@127.16.202.60:37597 shutdown complete.
I20260925 00:53:44.925050 23855 master_runner.cc:305] Clearing existing system tablet
I20260925 00:53:44.925279 23851 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: attempting to promote NON_VOTER bbde26b7bf7e4a4b9d1eeeeaa458ea34 to VOTER
I20260925 00:53:44.925290 23855 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.925345 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.925714 23851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } }
I20260925 00:53:44.926671 23855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 00:53:44.927482 23901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 LEARNER]: Refusing update from remote peer 34231b5a95474f3ebf353786e5000770: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 00:53:44.927506 24041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.927661 23855 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.927767 23855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 00:53:44.927798 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
uuid: "f0142bd70c61423082fa7eceb92b5fbc"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "7a715ceafc55751755249c57bb7f6b80"
server_key_iv: "0c74740319b9efdcda33ace8bbacee46"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.928146 23850 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 00:53:44.929715 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.927018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53090)
0925 00:53:44.929540 (+ 2522us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.929631 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2437}
I20260925 00:53:44.929924 23855 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:44.930594 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.930354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53100)
0925 00:53:44.930488 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.930532 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 00:53:44.930774 23278 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 -> Peer f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60:37597): Couldn't send request to peer f0142bd70c61423082fa7eceb92b5fbc. Status: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 00:53:44.931638 23850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bbde26b7bf7e4a4b9d1eeeeaa458ea34 (127.16.202.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.932266 23855 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 00:53:44.932348 23855 log.cc:1196] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 00:53:44.932559 23855 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Deleting consensus metadata
I20260925 00:53:44.932878 23855 master_runner.cc:315] Copying system tablet from 127.16.202.62:45099
I20260925 00:53:44.933130 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.933223 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:44.933861 24036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.933991 23855 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: tablet copy: Beginning tablet copy session from remote peer at address 127.16.202.62:45099
I20260925 00:53:44.934535 23901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bbde26b7bf7e4a4b9d1eeeeaa458ea34 (127.16.202.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.935878 24038 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.935994 24038 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:44.936192 24036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:44.943166 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.934660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.16.202.62:45099)
0925 00:53:44.934800 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:44.934804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:44.936954 (+ 2150us) server_negotiation.cc:407] Connection header received
0925 00:53:44.937087 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:44.937090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:44.937135 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:44.937199 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:44.937705 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.938330 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.939149 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.939377 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.939462 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.940460 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:44.940463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:44.940466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:44.940491 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:44.941739 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.942110 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.942114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.942115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.942186 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:44.942418 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:44.942422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:44.942423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:44.942590 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:44.942673 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:44.942917 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 00:53:44.943016 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:53:44.943184 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:44.934908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:46688)
0925 00:53:44.936873 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:44.936888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:44.937001 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:44.937227 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:44.937230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:44.937503 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:44.937595 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.937600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.938468 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.938471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.939041 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:44.939050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:44.939474 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:44.939478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:44.939557 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:44.940014 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:44.940028 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:44.940332 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:44.941840 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.941843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.941846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.942026 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.942262 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:44.942266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:44.942268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:44.942344 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:44.942672 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:44.942675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:44.942766 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 00:53:44.942855 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 00:53:44.942995 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1866,"thread_start_us":93,"threads_started":1}
I20260925 00:53:44.943809 23311 tablet_copy_service.cc:140] P 34231b5a95474f3ebf353786e5000770: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0142bd70c61423082fa7eceb92b5fbc ({username='oryx', principal='oryx/127.16.202.60@KRBTEST.COM'} at 127.0.0.1:46688)
I20260925 00:53:44.943890 23311 tablet_copy_service.cc:161] P 34231b5a95474f3ebf353786e5000770: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0142bd70c61423082fa7eceb92b5fbc at {username='oryx', principal='oryx/127.16.202.60@KRBTEST.COM'} at 127.0.0.1:46688: session id = f0142bd70c61423082fa7eceb92b5fbc-00000000000000000000000000000000
I20260925 00:53:44.944850 23311 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 00:53:44.945958 23855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 00:53:44.948578 23855 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: tablet copy: Starting download of 0 data blocks...
I20260925 00:53:44.948873 23855 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: tablet copy: Starting download of 1 WAL segments...
I20260925 00:53:44.950130 23855 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 00:53:44.951591 23311 tablet_copy_service.cc:342] P 34231b5a95474f3ebf353786e5000770: Request end of tablet copy session f0142bd70c61423082fa7eceb92b5fbc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.202.60@KRBTEST.COM'} at 127.0.0.1:46688
I20260925 00:53:44.951668 23311 tablet_copy_service.cc:433] P 34231b5a95474f3ebf353786e5000770: ending tablet copy session f0142bd70c61423082fa7eceb92b5fbc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0142bd70c61423082fa7eceb92b5fbc
I20260925 00:53:44.958326 23855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:44.960038 24051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.960038 24048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:44.960539 24049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:44.960578 23855 server_base.cc:1034] running on GCE node
I20260925 00:53:44.960884 23855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:44.961164 23855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:44.962291 23855 hybrid_clock.cc:648] HybridClock initialized: now 1790297624962286 us; error 27 us; skew 500 ppm
Sep 25 00:53:44 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297624, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:44.963631 23855 init.cc:377] Logged in from keytab as oryx/127.16.202.60@KRBTEST.COM (short username oryx)
I20260925 00:53:44.964092 23855 webserver.cc:460] Webserver started at http://127.16.202.60:46065/ using document root <none> and password file <none>
I20260925 00:53:44.964275 23855 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:44.964309 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:44.965564 23855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:44.966418 24058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:44.966621 23855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:44.966692 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
uuid: "f0142bd70c61423082fa7eceb92b5fbc"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "7a715ceafc55751755249c57bb7f6b80"
server_key_iv: "0c74740319b9efdcda33ace8bbacee46"
server_key_version: "encryptionkey@0"
I20260925 00:53:44.966777 23855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:44.974792 23855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:44.976567 23855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:44.980940 23855 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.60:37597
I20260925 00:53:44.981472 23855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
I20260925 00:53:44.981805 24110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.60:37597 every 8 connection(s)
I20260925 00:53:44.983350 24111 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:53:44.984200 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Bootstrap starting.
I20260925 00:53:44.993228 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:53:44.993494 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: Bootstrap complete.
I20260925 00:53:44.994016 24111 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } }
I20260925 00:53:44.994130 24111 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0142bd70c61423082fa7eceb92b5fbc, State: Initialized, Role: LEARNER
I20260925 00:53:44.994230 24111 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } }
I20260925 00:53:44.994539 24114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:44.994666 24111 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:44.994704 24114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: This master's current role is: LEARNER
I20260925 00:53:44.998891 24125 catalog_manager.cc:1259] Loaded cluster ID: cc3ac59429fa4eada86d966bc6f45633
I20260925 00:53:44.999047 24125 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: loading cluster ID for follower catalog manager: success
I20260925 00:53:45.001289 24125 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: acquiring CA information for follower catalog manager: success
I20260925 00:53:45.002897 24125 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:53:45.020789 23616 heartbeater.cc:503] Master 127.16.202.62:45099 was elected leader, sending a full tablet report...
W20260925 00:53:45.037999 24035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:45.038393 24035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:45.038457 24035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:45.044914 24035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:45.045037 24035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:45.045127 24035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:45.053352 24035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:43423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=46163
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:45.054740 24035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:45.055975 24035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:45.067217 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:45.067313 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:45.067173 24035 server_base.cc:1034] running on GCE node
W20260925 00:53:45.067148 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:45.067900 24035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:45.068637 24035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:45.069860 24035 hybrid_clock.cc:648] HybridClock initialized: now 1790297625069864 us; error 49 us; skew 500 ppm
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:45.073004 24035 init.cc:377] Logged in from keytab as oryx/127.16.202.1@KRBTEST.COM (short username oryx)
I20260925 00:53:45.074215 24035 webserver.cc:460] Webserver started at http://127.16.202.1:46163/ using document root <none> and password file <none>
I20260925 00:53:45.074958 24035 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:45.075070 24035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:45.080727 24035 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:53:45.083360 24140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.085091 24035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:45.085222 24035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "62c33483c2e14ceb81cdb603d4dcbce4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "92512f0a42ecfd0701e7c304ebbcff05"
server_key_iv: "bc7cf0ac5a6b921a86132b6fae439a74"
server_key_version: "encryptionkey@0"
I20260925 00:53:45.085958 24035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:45.106896 24035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:45.111188 24035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:45.111421 24035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:45.112085 24035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:45.113289 24035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:45.113354 24035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.113410 24035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:45.113437 24035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.123200 24035 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43423
I20260925 00:53:45.123229 24253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43423 every 8 connection(s)
I20260925 00:53:45.124228 24035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:45.128288 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 24035
I20260925 00:53:45.128505 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23487
I20260925 00:53:45.135957 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:39921
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:45099
--webserver_port=37059
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.60@KRBTEST.COM
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:45.143857 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.126116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:49755 (local address 127.16.202.61:42451)
0925 00:53:45.126253 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.126258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.127617 (+ 1359us) server_negotiation.cc:407] Connection header received
0925 00:53:45.129264 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.129267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.129333 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.129514 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.131576 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.132525 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.134700 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.135043 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.135122 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.139307 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.139311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.139313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.139357 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.141049 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.141771 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.141775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.141777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.141988 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142426 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.142429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.142431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.142658 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.142735 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.143542 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.143683 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:53:45.143865 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.128121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:52193 (local address 127.16.202.60:37597)
0925 00:53:45.130088 (+ 1967us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.130093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.130107 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:45.130150 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.130153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.130199 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.130284 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.131595 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.132854 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.134687 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.135062 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.135155 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.139088 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.139092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.139095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.139129 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.140976 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.141770 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.141774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.141776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.141887 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142423 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.142428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.142429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.142658 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.142730 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.143542 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.143683 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":84,"threads_started":1}
I20260925 00:53:45.145215 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.126522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.16.202.1:49755)
0925 00:53:45.127460 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.127507 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.128591 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.129883 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.129894 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.130452 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.131395 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.131418 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.132758 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.132762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.134558 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.134569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.135369 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.135373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.135664 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.136376 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.136398 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.139149 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.141150 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.141152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.141156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.141653 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.142077 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.142082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.142309 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.142800 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.142807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.143235 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.144008 (+ 773us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.144170 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":376,"spinlock_wait_cycles":27520,"thread_start_us":554,"threads_started":1}
I20260925 00:53:45.145466 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.128014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.16.202.1:52193)
0925 00:53:45.129951 (+ 1937us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.129967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.130064 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.130417 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.130420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.130623 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.131467 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.131474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.132991 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.132995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.134561 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.134568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.135333 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.135337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.135649 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.136623 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.136635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.138842 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.141126 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.141134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.141154 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.141653 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.142104 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.142108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.142309 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.142800 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.142807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.143235 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.144068 (+ 833us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.144228 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851,"mutex_wait_us":862,"spinlock_wait_cycles":37120,"thread_start_us":96,"threads_started":1}
I20260925 00:53:45.146095 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:48009 (local address 127.16.202.62:45099)
0925 00:53:45.128662 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.128666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.130306 (+ 1640us) server_negotiation.cc:407] Connection header received
0925 00:53:45.130483 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.130486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.130540 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.130594 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.131586 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.132560 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.134691 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.135055 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.135143 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.139366 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.139370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.139373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.139405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.141781 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142294 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.142308 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.142310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.142409 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142714 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.142717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.142719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.142947 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.143042 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.145789 (+ 2747us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.145932 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:53:45.146569 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.128384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:48009)
0925 00:53:45.130227 (+ 1843us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.130238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.130372 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.130738 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.130741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.130909 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.131395 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.131419 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.132758 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.132761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.134538 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.134551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.135333 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.135337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.135617 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.136509 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.136525 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.139181 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.141921 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.141925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.141927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.142168 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.142517 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.142520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.142522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.142609 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.143088 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.143091 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.143234 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.143462 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.143865 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1790,"mutex_wait_us":353,"spinlock_wait_cycles":6528,"thread_start_us":77,"threads_started":1}
I20260925 00:53:45.146870 24254 heartbeater.cc:347] Connected to a master server at 127.16.202.60:37597
I20260925 00:53:45.147126 24254 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.147297 24255 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45099
I20260925 00:53:45.147387 24255 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.147799 24254 heartbeater.cc:511] Master 127.16.202.60:37597 requested a full tablet report, sending...
I20260925 00:53:45.147799 24255 heartbeater.cc:511] Master 127.16.202.62:45099 requested a full tablet report, sending...
I20260925 00:53:45.148336 24256 heartbeater.cc:347] Connected to a master server at 127.16.202.61:42451
I20260925 00:53:45.148408 24256 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.148562 24256 heartbeater.cc:511] Master 127.16.202.61:42451 requested a full tablet report, sending...
I20260925 00:53:45.149396 24075 ts_manager.cc:194] Registered new tserver with Master: 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423)
I20260925 00:53:45.150099 23287 ts_manager.cc:194] Re-registered known tserver with Master: 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423)
I20260925 00:53:45.150862 23287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:48009
I20260925 00:53:45.151914 23891 ts_manager.cc:194] Registered new tserver with Master: 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423)
I20260925 00:53:45.239454 23750 heartbeater.cc:503] Master 127.16.202.62:45099 was elected leader, sending a full tablet report...
W20260925 00:53:45.255177 24262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:45.255507 24262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:45.255576 24262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:45.259764 24262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:45.259860 24262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:45.259956 24262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:45.265553 24262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:39921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=37059
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:45.267009 24262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:45.268162 24262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:45.278009 24270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:45.278009 24268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:45.278004 24267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:45.278264 24262 server_base.cc:1034] running on GCE node
I20260925 00:53:45.278700 24262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:45.279385 24262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:45.280576 24262 hybrid_clock.cc:648] HybridClock initialized: now 1790297625280548 us; error 40 us; skew 500 ppm
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:45.283743 24262 init.cc:377] Logged in from keytab as oryx/127.16.202.2@KRBTEST.COM (short username oryx)
I20260925 00:53:45.284942 24262 webserver.cc:460] Webserver started at http://127.16.202.2:37059/ using document root <none> and password file <none>
I20260925 00:53:45.285575 24262 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:45.285672 24262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:45.290213 24262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 00:53:45.293048 24277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.294648 24262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:53:45.294786 24262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "ee1795c71d5741c68314cbac9a9efea4"
format_stamp: "Formatted at 2026-09-25 00:53:43 on dist-test-slave-rpk1"
server_key: "25207c1b2d5bc16e10f5c48f36985666"
server_key_iv: "42d5df7532022b522f2a3589a016d898"
server_key_version: "encryptionkey@0"
I20260925 00:53:45.295507 24262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:45.317585 24262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:45.321048 24262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:45.321242 24262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:45.321853 24262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:45.322722 24262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:45.322775 24262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.322820 24262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:45.322849 24262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.331771 24262 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:39921
I20260925 00:53:45.331797 24390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:39921 every 8 connection(s)
I20260925 00:53:45.333166 24262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:45.333566 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 24262
I20260925 00:53:45.333741 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23621
I20260925 00:53:45.341183 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:44973
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:45099
--webserver_port=40565
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.60@KRBTEST.COM
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
I20260925 00:53:45.352788 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.338251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:52707 (local address 127.16.202.61:42451)
0925 00:53:45.338354 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.338358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.338976 (+ 618us) server_negotiation.cc:407] Connection header received
0925 00:53:45.339131 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.339134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.339183 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.339233 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.340907 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.341733 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.343705 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.344049 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.344131 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.349449 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.349453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.349456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.349488 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.351022 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351463 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.351466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.351468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.351544 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351809 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.351813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.351814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.352034 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.352100 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.352451 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.352579 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:53:45.352788 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.335543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:45681 (local address 127.16.202.60:37597)
0925 00:53:45.335817 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.335822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.337352 (+ 1530us) server_negotiation.cc:407] Connection header received
0925 00:53:45.338652 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.338655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.338714 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.338767 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.340922 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.341734 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.343706 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.344085 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.344149 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.348881 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.348886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.348889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.348922 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.350732 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351191 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.351194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.351196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.351277 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351581 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.351584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.351585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.351801 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.351871 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.352468 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.352585 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:53:45.352837 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.337906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:50591 (local address 127.16.202.62:45099)
0925 00:53:45.338025 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.338030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.338551 (+ 521us) server_negotiation.cc:407] Connection header received
0925 00:53:45.338653 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.338656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.338714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.338813 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.340913 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.341635 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.343728 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.344082 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.344162 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.348018 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.348024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.348026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.348063 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.350100 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.350831 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.350836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.350838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.350936 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351371 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.351375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.351377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.351627 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.351703 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.352479 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.352624 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:53:45.354847 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.337757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:50591)
0925 00:53:45.338451 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.338465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.338570 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.339159 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.339166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.339870 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.340728 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.340741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.341826 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.341829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.343563 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.343575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.344358 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.344361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.344669 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.345484 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.345503 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.347718 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.350264 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.350272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.350295 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.350686 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.351058 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.351063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.351258 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.351800 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.351808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.352180 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.353152 (+ 972us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.353728 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":477,"spinlock_wait_cycles":96896,"thread_start_us":84,"threads_started":1}
I20260925 00:53:45.354848 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.335987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.16.202.2:45681)
0925 00:53:45.337165 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.337226 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.338378 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.339155 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.339165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.339860 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.340715 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.340739 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.341881 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.341884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.343565 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.343572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.344358 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.344362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.344702 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.345614 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.345627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.348657 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.350866 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.350869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.350872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.351084 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.351392 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.351397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.351486 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.351924 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.351927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.352173 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.353152 (+ 979us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.353635 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":345,"spinlock_wait_cycles":10112,"thread_start_us":683,"threads_started":1}
I20260925 00:53:45.354847 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.338160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.16.202.2:52707)
0925 00:53:45.338901 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.338914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.339035 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.339384 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.339397 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.339870 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.340715 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.340739 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.341881 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.341884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.343542 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.343556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.344358 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.344362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.344705 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.345523 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.345538 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.349279 (+ 3741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.351138 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.351143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.351362 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.351643 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.351646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.351648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.351716 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.352164 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.352167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.352243 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.353656 (+ 1413us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.353777 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":439,"spinlock_wait_cycles":30592,"thread_start_us":81,"threads_started":1}
I20260925 00:53:45.356144 24391 heartbeater.cc:347] Connected to a master server at 127.16.202.60:37597
I20260925 00:53:45.356240 24393 heartbeater.cc:347] Connected to a master server at 127.16.202.61:42451
I20260925 00:53:45.356249 24392 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45099
I20260925 00:53:45.356400 24392 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.356400 24393 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.356400 24391 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.357085 24392 heartbeater.cc:511] Master 127.16.202.62:45099 requested a full tablet report, sending...
I20260925 00:53:45.357079 24393 heartbeater.cc:511] Master 127.16.202.61:42451 requested a full tablet report, sending...
I20260925 00:53:45.357079 24391 heartbeater.cc:511] Master 127.16.202.60:37597 requested a full tablet report, sending...
I20260925 00:53:45.358213 23287 ts_manager.cc:194] Re-registered known tserver with Master: ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921)
I20260925 00:53:45.358292 23891 ts_manager.cc:194] Registered new tserver with Master: ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921)
I20260925 00:53:45.358292 24075 ts_manager.cc:194] Registered new tserver with Master: ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921)
I20260925 00:53:45.358991 23287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:50591
I20260925 00:53:45.440076 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.435523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53102)
0925 00:53:45.435822 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.435835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.436054 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.436383 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.436387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.436518 (+ 131us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:45.436636 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.436645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.437779 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.437782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.438711 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.438724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.439658 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.439662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.439748 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.439773 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.439842 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.439944 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260925 00:53:45.440076 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.435644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.16.202.60:37597)
0925 00:53:45.435779 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.435783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.436005 (+ 222us) server_negotiation.cc:407] Connection header received
0925 00:53:45.436174 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.436177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.436244 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.436362 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:45.436790 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.437638 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.438854 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.439494 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.439618 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.439794 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.439860 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.439947 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:53:45.441079 24085 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 00:53:45.441511 24114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34231b5a95474f3ebf353786e5000770. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:45.441651 24114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: This master's current role is: LEARNER
W20260925 00:53:45.463297 24398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:45.463608 24398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:45.463650 24398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:45.468202 24398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:45.468276 24398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:45.468346 24398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:45.474040 24398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:44973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=40565
--tserver_master_addrs=127.16.202.62:45099,127.16.202.61:42451,127.16.202.60:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:45.475190 24398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:45.476276 24398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:53:45.486974 24398 server_base.cc:1034] running on GCE node
W20260925 00:53:45.486932 24404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:45.486932 24405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:45.486927 24407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:45.487589 24398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:45.488240 24398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:45.489408 24398 hybrid_clock.cc:648] HybridClock initialized: now 1790297625489381 us; error 35 us; skew 500 ppm
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:45.492106 24398 init.cc:377] Logged in from keytab as oryx/127.16.202.3@KRBTEST.COM (short username oryx)
I20260925 00:53:45.493085 24398 webserver.cc:460] Webserver started at http://127.16.202.3:40565/ using document root <none> and password file <none>
I20260925 00:53:45.493629 24398 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:45.493710 24398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:45.497732 24398 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:53:45.499842 24414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.501191 24398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:45.501297 24398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "0719fa59798645649e691f3a6c185032"
format_stamp: "Formatted at 2026-09-25 00:53:44 on dist-test-slave-rpk1"
server_key: "32c56a1d549a5c3f7d6328e722f9a71d"
server_key_iv: "e817f96c846e821f11d0f7288dfe1f07"
server_key_version: "encryptionkey@0"
I20260925 00:53:45.501825 24398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:45.514585 24398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:45.517748 24398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:45.517949 24398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:45.518567 24398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:45.519519 24398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:45.519567 24398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.519622 24398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:45.519650 24398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:45.528594 24398 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:44973
I20260925 00:53:45.528654 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:44973 every 8 connection(s)
I20260925 00:53:45.529685 24398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:45.540016 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 24398
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.60@KRBTEST.COM
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
Sep 25 00:53:45 dist-test-slave-rpk1 krb5kdc[23246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297625, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.61@KRBTEST.COM
I20260925 00:53:45.545745 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.531558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:49443 (local address 127.16.202.62:45099)
0925 00:53:45.531708 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.531713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.533080 (+ 1367us) server_negotiation.cc:407] Connection header received
0925 00:53:45.534157 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.534161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.534218 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.534297 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.537412 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.538244 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.539306 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.539640 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.539760 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.541831 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.541835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.541838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.541869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.543568 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.544035 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.544039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.544041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.544133 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.544440 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.544443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.544445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.544668 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.544780 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.545448 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.545578 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:45.546531 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.534039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:60847 (local address 127.16.202.61:42451)
0925 00:53:45.534464 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.534468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.534480 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:53:45.534517 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.534519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.534554 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.534619 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.536710 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.537253 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.538522 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.538857 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.538936 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.543237 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.543242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.543244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.543275 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.544869 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.545320 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.545324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.545325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.545421 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.545688 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.545691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.545693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.545881 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.545967 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.546275 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.546397 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
I20260925 00:53:45.546857 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.533362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.16.202.3:57325)
0925 00:53:45.533627 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.533640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.533917 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.534489 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.534498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.535077 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.535787 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.535793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.536790 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.536801 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.537801 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.537814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.538517 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.538522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.538716 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.539322 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.539344 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.541310 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.543155 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.543161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.543183 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.543442 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.543987 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.543990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.544000 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.544197 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.544731 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.544738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.545147 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.545358 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.545721 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":81,"spinlock_wait_cycles":17280,"thread_start_us":79,"threads_started":1}
I20260925 00:53:45.546864 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.533689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.16.202.3:60847)
0925 00:53:45.533889 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.533902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.533987 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.534655 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.534658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.535070 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.535775 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.535791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.537360 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.537364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.538401 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.538409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.539087 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.539090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.539188 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.541282 (+ 2094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.541296 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.543083 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.544994 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.544997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.544999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.545213 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.545512 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.545514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.545516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.545593 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.546005 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.546008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.546087 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.546177 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.546334 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":88,"thread_start_us":70,"threads_started":1}
I20260925 00:53:45.547554 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.533550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:57325 (local address 127.16.202.60:37597)
0925 00:53:45.533671 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.533675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.533690 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:45.534246 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.534250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.534304 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.534374 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.535929 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.536552 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.538084 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.538324 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.538458 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.541519 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.541524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.541527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.541557 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.543024 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.543754 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.543758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.543760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.543860 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.544310 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.544314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.544316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.544554 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.544649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.547276 (+ 2627us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.547409 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:45.548230 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.532001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:49443)
0925 00:53:45.532917 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.532965 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.533917 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.534489 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.534498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.535123 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:45.535785 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.535793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.538398 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.538402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.539191 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.539199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.539787 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.539791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.539879 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.540382 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:45.540401 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:45.541666 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:45.543664 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.543667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.543669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.543908 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.544247 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:45.544249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:45.544251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:45.544332 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:45.544852 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:45.544855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:45.545149 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.546164 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.547500 (+ 1336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":134,"spinlock_wait_cycles":53376,"thread_start_us":519,"threads_started":1}
I20260925 00:53:45.548377 24530 heartbeater.cc:347] Connected to a master server at 127.16.202.61:42451
I20260925 00:53:45.548727 24528 heartbeater.cc:347] Connected to a master server at 127.16.202.60:37597
I20260925 00:53:45.548820 24528 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.549479 24528 heartbeater.cc:511] Master 127.16.202.60:37597 requested a full tablet report, sending...
I20260925 00:53:45.549594 24529 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45099
I20260925 00:53:45.549670 24529 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.549827 24529 heartbeater.cc:511] Master 127.16.202.62:45099 requested a full tablet report, sending...
I20260925 00:53:45.548732 24530 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:45.550170 24530 heartbeater.cc:511] Master 127.16.202.61:42451 requested a full tablet report, sending...
I20260925 00:53:45.551301 23287 ts_manager.cc:194] Re-registered known tserver with Master: 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973)
I20260925 00:53:45.552915 24075 ts_manager.cc:194] Registered new tserver with Master: 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973)
I20260925 00:53:45.553580 23891 ts_manager.cc:194] Registered new tserver with Master: 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973)
I20260925 00:53:45.553862 23287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:49443
I20260925 00:53:45.555846 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.542555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.16.202.62:45099)
0925 00:53:45.542883 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.542887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.545835 (+ 2948us) server_negotiation.cc:407] Connection header received
0925 00:53:45.545877 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.545880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.545927 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.545994 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.546957 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.547787 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.549460 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.549786 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.549888 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.551738 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.551741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.551744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.551772 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.553583 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.554275 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.554279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.554280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.554376 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.554683 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.554686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.554688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.554902 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.555032 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.555533 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.555661 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260925 00:53:45.557072 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.544265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.16.202.61:42451)
0925 00:53:45.546677 (+ 2412us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.546681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.548279 (+ 1598us) server_negotiation.cc:407] Connection header received
0925 00:53:45.548399 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.548402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.548455 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.548525 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.549480 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.550203 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.551266 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.551538 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.551610 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.553342 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.553347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.553349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.553373 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.555216 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.555759 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.555762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.555764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.555849 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.556142 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.556145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.556147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.556340 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.556447 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.556816 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.556919 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2359,"thread_start_us":77,"threads_started":1}
I20260925 00:53:45.559806 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.544986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.16.202.60:37597)
0925 00:53:45.545480 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.545483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.550545 (+ 5062us) server_negotiation.cc:407] Connection header received
0925 00:53:45.550654 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.550658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.550710 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.550788 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.551691 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.552374 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.553771 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.554194 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.554265 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.555778 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.555782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.555784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.555808 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.558252 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.558697 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.558701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.558703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.558782 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.559047 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.559049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.559050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.559216 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.559311 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.559553 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.559656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":64,"threads_started":1}
I20260925 00:53:45.559101 23287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:45.562587 23287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:45.565306 24547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: NON_VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: true } } }
I20260925 00:53:45.565448 24547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:45.569576 24547 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: attempting to promote NON_VOTER f0142bd70c61423082fa7eceb92b5fbc to VOTER
I20260925 00:53:45.569916 24547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } }
I20260925 00:53:45.571123 24085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 LEARNER]: Refusing update from remote peer 34231b5a95474f3ebf353786e5000770: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 00:53:45.571218 23901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Refusing update from remote peer 34231b5a95474f3ebf353786e5000770: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 00:53:45.571524 24039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 00:53:45.571715 24546 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 00:53:45.573884 24547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.574309 23901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.574352 24085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.575451 24547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.575555 24547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:45.575680 24547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.575703 24546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.575753 24547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:45.575945 24114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.576045 24114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:45.575770 24546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:45.578687 24555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34231b5a95474f3ebf353786e5000770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:45.578817 24555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:45.582824 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.577836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:39921 (local address 127.0.0.1:33422)
0925 00:53:45.577962 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.577977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.578055 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.578929 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.578933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.578952 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:45.579060 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.579067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.580377 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.580380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.581340 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.581347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.582393 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.582405 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.582493 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.582518 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.582598 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.582688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 00:53:45.582996 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.578030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.16.202.2:39921)
0925 00:53:45.578377 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.578389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.578412 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:53:45.578493 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.578499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.578754 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.579216 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:45.579353 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.580215 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.581440 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.582232 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.582387 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.582532 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.582753 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.582859 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":154,"threads_started":1}
I20260925 00:53:45.584903 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.578516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43423 (local address 127.0.0.1:52682)
0925 00:53:45.580437 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.580448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.580528 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.581045 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.581048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.581067 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:45.581153 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.581158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.582193 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.582196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.583221 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.583231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.584406 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.584409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.584473 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.584491 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.584555 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.584627 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1865,"thread_start_us":85,"threads_started":1}
I20260925 00:53:45.585139 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.578905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.16.202.1:43423)
0925 00:53:45.579196 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.579208 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.580541 (+ 1333us) server_negotiation.cc:407] Connection header received
0925 00:53:45.580630 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.580635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.580901 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.581057 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:45.581265 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.582091 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.583321 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.584095 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.584197 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.584351 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.584605 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.584732 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
I20260925 00:53:45.586726 24325 tablet_service.cc:1467] Processing CreateTablet for tablet ff59bbd2a1df4003900fb10140f7df25 (DEFAULT_TABLE table=test-table [id=34ae294af6014231960d69f18239d1dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:45.587721 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59bbd2a1df4003900fb10140f7df25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:45.587872 24188 tablet_service.cc:1467] Processing CreateTablet for tablet ff59bbd2a1df4003900fb10140f7df25 (DEFAULT_TABLE table=test-table [id=34ae294af6014231960d69f18239d1dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:45.588838 24188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59bbd2a1df4003900fb10140f7df25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:45.595705 24561 tablet_bootstrap.cc:492] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Bootstrap starting.
I20260925 00:53:45.596335 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.578728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.16.202.3:44973)
0925 00:53:45.580803 (+ 2075us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.580813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.580834 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:53:45.580908 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.580913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.581085 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.581211 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:45.581571 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.583292 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.584246 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.586107 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.586190 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.586318 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.596024 (+ 9706us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.596171 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":84,"threads_started":1}
I20260925 00:53:45.596936 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.578331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:44973 (local address 127.0.0.1:34256)
0925 00:53:45.579545 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.579558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.579641 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.581347 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.581350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.581368 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:45.581454 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.581459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.583378 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.583381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.584127 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.584136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.595657 (+ 11521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.595662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.595792 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.595825 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.596622 (+ 797us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.596774 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"thread_start_us":105,"threads_started":1}
I20260925 00:53:45.599565 24561 tablet_bootstrap.cc:654] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:45.600442 24561 log.cc:824] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:45.600288 24462 tablet_service.cc:1467] Processing CreateTablet for tablet ff59bbd2a1df4003900fb10140f7df25 (DEFAULT_TABLE table=test-table [id=34ae294af6014231960d69f18239d1dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:45.601480 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59bbd2a1df4003900fb10140f7df25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:45.601706 24562 tablet_bootstrap.cc:492] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Bootstrap starting.
I20260925 00:53:45.603629 24561 tablet_bootstrap.cc:492] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: No bootstrap required, opened a new log
I20260925 00:53:45.603842 24561 ts_tablet_manager.cc:1397] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 00:53:45.604123 24562 tablet_bootstrap.cc:654] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:45.604997 24562 log.cc:824] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:45.607088 24562 tablet_bootstrap.cc:492] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: No bootstrap required, opened a new log
I20260925 00:53:45.607292 24562 ts_tablet_manager.cc:1397] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:53:45.608740 24561 raft_consensus.cc:348] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.609050 24561 raft_consensus.cc:374] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:45.609107 24561 raft_consensus.cc:729] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1795c71d5741c68314cbac9a9efea4, State: Initialized, Role: FOLLOWER
I20260925 00:53:45.609593 24561 consensus_queue.cc:260] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.610371 24392 heartbeater.cc:503] Master 127.16.202.62:45099 was elected leader, sending a full tablet report...
I20260925 00:53:45.612488 24566 tablet_bootstrap.cc:492] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Bootstrap starting.
I20260925 00:53:45.611533 24562 raft_consensus.cc:348] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.613834 24562 raft_consensus.cc:374] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:45.613924 24562 raft_consensus.cc:729] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c33483c2e14ceb81cdb603d4dcbce4, State: Initialized, Role: FOLLOWER
I20260925 00:53:45.614418 24562 consensus_queue.cc:260] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.615116 24561 ts_tablet_manager.cc:1428] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260925 00:53:45.615432 24566 tablet_bootstrap.cc:654] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:45.615554 24562 ts_tablet_manager.cc:1428] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 00:53:45.615753 24255 heartbeater.cc:503] Master 127.16.202.62:45099 was elected leader, sending a full tablet report...
I20260925 00:53:45.616422 24566 log.cc:824] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:45.618906 24566 tablet_bootstrap.cc:492] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: No bootstrap required, opened a new log
I20260925 00:53:45.619133 24566 ts_tablet_manager.cc:1397] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 00:53:45.622937 24566 raft_consensus.cc:348] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.623174 24566 raft_consensus.cc:374] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:45.623221 24566 raft_consensus.cc:729] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0719fa59798645649e691f3a6c185032, State: Initialized, Role: FOLLOWER
I20260925 00:53:45.623643 24566 consensus_queue.cc:260] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.624713 24566 ts_tablet_manager.cc:1428] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260925 00:53:45.624945 24529 heartbeater.cc:503] Master 127.16.202.62:45099 was elected leader, sending a full tablet report...
W20260925 00:53:45.626314 24257 tablet.cc:2369] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:45.679021 24569 raft_consensus.cc:482] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:45.679356 24569 raft_consensus.cc:504] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.680547 24569 leader_election.cc:288] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921), 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423)
I20260925 00:53:45.684593 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.680883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43423 (local address 127.16.202.3:53333)
0925 00:53:45.681002 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.681019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.681125 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.681473 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.681476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.681499 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:45.681633 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.681640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.682450 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.682454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.683377 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.683412 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.684136 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.684140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.684254 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.684279 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.684360 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.684442 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":48}
I20260925 00:53:45.684713 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.680952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:53333 (local address 127.16.202.1:43423)
0925 00:53:45.681067 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.681070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.681084 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:45.681297 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.681300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.681355 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.681458 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:45.681745 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.682305 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.683528 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.684014 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.684110 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.684154 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.684459 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.684546 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:45.684839 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.680890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:39921 (local address 127.16.202.3:38139)
0925 00:53:45.681361 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:45.681374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:45.681459 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:45.681704 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:45.681707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:45.681724 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:45.681831 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.681837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.682663 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.682667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.683653 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:45.683663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.684421 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.684425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:45.684528 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.684551 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:45.684636 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 00:53:45.684731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":314}
I20260925 00:53:45.684872 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.680961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:38139 (local address 127.16.202.2:39921)
0925 00:53:45.681079 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.681084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.681463 (+ 379us) server_negotiation.cc:407] Connection header received
0925 00:53:45.681544 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.681547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.681597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.681687 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:45.681982 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.682549 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.683806 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.684278 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.684419 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.684465 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.684661 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.684762 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:45.685380 24208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59bbd2a1df4003900fb10140f7df25" candidate_uuid: "0719fa59798645649e691f3a6c185032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" is_pre_election: true
I20260925 00:53:45.685724 24208 raft_consensus.cc:2454] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0719fa59798645649e691f3a6c185032 in term 0.
I20260925 00:53:45.685829 24345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59bbd2a1df4003900fb10140f7df25" candidate_uuid: "0719fa59798645649e691f3a6c185032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1795c71d5741c68314cbac9a9efea4" is_pre_election: true
I20260925 00:53:45.686185 24417 leader_election.cc:302] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0719fa59798645649e691f3a6c185032, 62c33483c2e14ceb81cdb603d4dcbce4; no voters:
I20260925 00:53:45.686486 24569 raft_consensus.cc:2781] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:45.686553 24569 raft_consensus.cc:482] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:45.686580 24569 raft_consensus.cc:3037] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:45.686152 24345 raft_consensus.cc:2454] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0719fa59798645649e691f3a6c185032 in term 0.
I20260925 00:53:45.687748 24569 raft_consensus.cc:504] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.688118 24569 leader_election.cc:288] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [CANDIDATE]: Term 1 election: Requested vote from peers ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921), 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423)
I20260925 00:53:45.688536 24208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59bbd2a1df4003900fb10140f7df25" candidate_uuid: "0719fa59798645649e691f3a6c185032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c33483c2e14ceb81cdb603d4dcbce4"
I20260925 00:53:45.688560 24345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59bbd2a1df4003900fb10140f7df25" candidate_uuid: "0719fa59798645649e691f3a6c185032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1795c71d5741c68314cbac9a9efea4"
I20260925 00:53:45.688701 24345 raft_consensus.cc:3037] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:45.688702 24208 raft_consensus.cc:3037] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:45.689989 24208 raft_consensus.cc:2454] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0719fa59798645649e691f3a6c185032 in term 1.
I20260925 00:53:45.689988 24345 raft_consensus.cc:2454] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0719fa59798645649e691f3a6c185032 in term 1.
I20260925 00:53:45.690353 24417 leader_election.cc:302] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0719fa59798645649e691f3a6c185032, 62c33483c2e14ceb81cdb603d4dcbce4, ee1795c71d5741c68314cbac9a9efea4; no voters:
I20260925 00:53:45.690655 24569 raft_consensus.cc:2781] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:45.690968 24569 raft_consensus.cc:686] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 1 LEADER]: Becoming Leader. State: Replica: 0719fa59798645649e691f3a6c185032, State: Running, Role: LEADER
I20260925 00:53:45.691437 24569 consensus_queue.cc:237] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:45.694973 23291 catalog_manager.cc:5495] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0719fa59798645649e691f3a6c185032 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "0719fa59798645649e691f3a6c185032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } health_report { overall_health: HEALTHY } } }
I20260925 00:53:45.708503 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.704645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.16.202.3:44973)
0925 00:53:45.704781 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.704785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.704799 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:45.704984 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.704987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.705047 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.705140 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:45.705522 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.706056 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.706975 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.707194 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.707283 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.707608 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:45.708058 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:45.708165 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.708317 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.708382 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:53:45.714953 24345 raft_consensus.cc:1261] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Refusing update from remote peer 0719fa59798645649e691f3a6c185032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:45.714953 24208 raft_consensus.cc:1261] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 FOLLOWER]: Refusing update from remote peer 0719fa59798645649e691f3a6c185032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:45.715873 24569 consensus_queue.cc:1035] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:45.716132 24570 consensus_queue.cc:1035] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:45.726058 24573 mvcc.cc:205] Tried to move back new op lower bound from 7333059074920181760 to 7333059074835038208. Current Snapshot: MvccSnapshot[applied={T|T < 7333059074920181760}]
I20260925 00:53:45.730290 24575 mvcc.cc:205] Tried to move back new op lower bound from 7333059074920181760 to 7333059074835038208. Current Snapshot: MvccSnapshot[applied={T|T < 7333059074920181760}]
I20260925 00:53:45.739035 24574 mvcc.cc:205] Tried to move back new op lower bound from 7333059074920181760 to 7333059074835038208. Current Snapshot: MvccSnapshot[applied={T|T < 7333059074920181760}]
I20260925 00:53:45.739910 23291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:45.740074 23291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:45.745188 23291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34231b5a95474f3ebf353786e5000770: Sending DeleteTablet for 3 replicas of tablet ff59bbd2a1df4003900fb10140f7df25
I20260925 00:53:45.746385 24325 tablet_service.cc:1514] Processing DeleteTablet for tablet ff59bbd2a1df4003900fb10140f7df25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:45 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33422
I20260925 00:53:45.746657 24462 tablet_service.cc:1514] Processing DeleteTablet for tablet ff59bbd2a1df4003900fb10140f7df25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:45 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:34256
I20260925 00:53:45.747169 24188 tablet_service.cc:1514] Processing DeleteTablet for tablet ff59bbd2a1df4003900fb10140f7df25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:45 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:52682
I20260925 00:53:45.747272 24584 tablet_replica.cc:330] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: stopping tablet replica
I20260925 00:53:45.747694 24584 raft_consensus.cc:2229] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:45.748064 24584 raft_consensus.cc:2258] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:45.748354 24583 tablet_replica.cc:330] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: stopping tablet replica
I20260925 00:53:45.748773 24583 raft_consensus.cc:2229] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:45.749163 24583 raft_consensus.cc:2258] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:45.750191 24584 ts_tablet_manager.cc:1905] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:45.750705 24583 ts_tablet_manager.cc:1905] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:45.752043 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23259
I20260925 00:53:45.753031 24584 ts_tablet_manager.cc:1918] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:45.753120 24584 log.cc:1196] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/ff59bbd2a1df4003900fb10140f7df25
I20260925 00:53:45.753429 24584 ts_tablet_manager.cc:1939] T ff59bbd2a1df4003900fb10140f7df25 P 0719fa59798645649e691f3a6c185032: Deleting consensus metadata
I20260925 00:53:45.754436 24583 ts_tablet_manager.cc:1918] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:45.754537 24583 log.cc:1196] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/ff59bbd2a1df4003900fb10140f7df25
I20260925 00:53:45.754842 24583 ts_tablet_manager.cc:1939] T ff59bbd2a1df4003900fb10140f7df25 P ee1795c71d5741c68314cbac9a9efea4: Deleting consensus metadata
I20260925 00:53:45.757282 24585 tablet_replica.cc:330] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: stopping tablet replica
I20260925 00:53:45.757759 24585 raft_consensus.cc:2229] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:45.758237 24585 raft_consensus.cc:2258] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 00:53:45.759852 24418 connection.cc:535] server connection from 127.0.0.1:34256 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:53:45.760052 24585 ts_tablet_manager.cc:1905] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:53:45.760474 24278 connection.cc:535] server connection from 127.0.0.1:33422 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:53:45.763521 24585 ts_tablet_manager.cc:1918] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:45.763726 24585 log.cc:1196] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/ff59bbd2a1df4003900fb10140f7df25
I20260925 00:53:45.764146 24585 ts_tablet_manager.cc:1939] T ff59bbd2a1df4003900fb10140f7df25 P 62c33483c2e14ceb81cdb603d4dcbce4: Deleting consensus metadata
W20260925 00:53:45.765558 24144 connection.cc:441] server connection from 127.0.0.1:52682 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52682 (request call id 1) could send its response
I20260925 00:53:45.768366 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.749049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.16.202.61:42451)
0925 00:53:45.749283 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.749287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.752862 (+ 3575us) server_negotiation.cc:407] Connection header received
0925 00:53:45.757272 (+ 4410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.757278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.757355 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.757459 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.758568 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.759397 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.760648 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.761044 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.761121 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.763918 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.763923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.763926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.763961 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.766013 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.766508 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.766512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.766514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.766604 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.766928 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.766932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.766933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.767251 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.767634 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.767987 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.768178 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260925 00:53:45.777225 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.765604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.16.202.61:42451)
0925 00:53:45.765735 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.765740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.766238 (+ 498us) server_negotiation.cc:407] Connection header received
0925 00:53:45.766432 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.766435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.766489 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.766549 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.767571 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.768777 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.770002 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.770368 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.770459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.772304 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.772309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.772312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.772353 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.774800 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.775360 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.775364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.775366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.775486 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.775863 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.775867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.775869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.776115 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.776247 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.776881 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.777051 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:53:45.778481 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.766157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.16.202.60:37597)
0925 00:53:45.766292 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.766296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.767785 (+ 1489us) server_negotiation.cc:407] Connection header received
0925 00:53:45.768004 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.768007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.768059 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.768113 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.769995 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.770752 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.772002 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.772320 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.772387 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.774309 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.774313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.774316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.774361 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.776424 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.777055 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.777059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.777061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.777182 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.777498 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.777501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.777503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.777718 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.777835 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.778165 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.778307 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":12416}
I20260925 00:53:45.778671 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:45.749157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.16.202.60:37597)
0925 00:53:45.749373 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:45.749377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:45.769285 (+ 19908us) server_negotiation.cc:407] Connection header received
0925 00:53:45.769463 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:45.769466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:45.769519 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:45.769564 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:45.770388 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.771192 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.772242 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:45.772548 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:45.772611 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:45.774309 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:45.774313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:45.774316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:45.774359 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:45.776445 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.777055 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.777059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.777061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.777192 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:45.777824 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:45.777827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:45.777829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:45.778025 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:45.778122 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:45.778426 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 00:53:45.778542 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"spinlock_wait_cycles":9728}
I20260925 00:53:46.734194 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:46.733752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:58739)
0925 00:53:46.733992 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:46.734084 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
W20260925 00:53:46.734506 24392 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:45099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:46.738070 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:46.737644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:33669)
0925 00:53:46.737878 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:46.737968 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
W20260925 00:53:46.738431 24529 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:45099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:46.751997 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:46.751540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:45355)
0925 00:53:46.751797 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:46.751901 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260925 00:53:46.752295 24255 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:45099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:47.245134 24596 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34231b5a95474f3ebf353786e5000770)
I20260925 00:53:47.245401 24596 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } }
I20260925 00:53:47.246452 24596 leader_election.cc:288] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099), f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60:37597)
I20260925 00:53:47.247071 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.246633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47162)
0925 00:53:47.246881 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.246965 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260925 00:53:47.247275 24599 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34231b5a95474f3ebf353786e5000770)
W20260925 00:53:47.247292 23878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:47.247399 24599 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } }
I20260925 00:53:47.247941 24599 leader_election.cc:288] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099), bbde26b7bf7e4a4b9d1eeeeaa458ea34 (127.16.202.61:42451)
I20260925 00:53:47.248229 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.247913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47178)
0925 00:53:47.248110 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.248161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260925 00:53:47.248330 23878 leader_election.cc:334] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099): Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:47.248790 24603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.248050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47198)
0925 00:53:47.248596 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.248699 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":60,"threads_started":1}
W20260925 00:53:47.248963 24062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:47.250242 24603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.250009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47204)
0925 00:53:47.250125 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.250178 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 00:53:47.250430 24062 leader_election.cc:334] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099): Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:47.251899 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.247537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.16.202.60:37597)
0925 00:53:47.247736 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.247740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.247754 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:47.247807 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.247810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.247859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.248234 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.248333 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.249656 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.250924 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.251383 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.251473 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.251534 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.251698 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.251772 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260925 00:53:47.251905 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.247035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37597 (local address 127.0.0.1:53156)
0925 00:53:47.247201 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.247215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.247318 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.247978 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.247982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.248002 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.248131 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.248138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.249775 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.249779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.250808 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.250818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.251494 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.251497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.251591 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.251610 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.251674 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.251794 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":43,"thread_start_us":70,"threads_started":1}
I20260925 00:53:47.252429 24085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0142bd70c61423082fa7eceb92b5fbc" is_pre_election: true
I20260925 00:53:47.252641 24085 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbde26b7bf7e4a4b9d1eeeeaa458ea34 in term 1.
I20260925 00:53:47.252774 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.248353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:42451 (local address 127.0.0.1:46324)
0925 00:53:47.249179 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.249192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.249304 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.249598 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.249601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.249615 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.249690 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.249695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.250633 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.250637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.251495 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.251505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.252212 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.252215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.252420 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.252448 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.252520 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.252591 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":25,"thread_start_us":61,"threads_started":1}
I20260925 00:53:47.252827 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.248487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.16.202.61:42451)
0925 00:53:47.249051 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.249059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.249249 (+ 190us) server_negotiation.cc:407] Connection header received
0925 00:53:47.249405 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.249408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.249444 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.249492 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.249790 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.250526 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.251617 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.252095 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.252207 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.252282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.252565 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.252647 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":511,"thread_start_us":111,"threads_started":1}
I20260925 00:53:47.253180 23878 leader_election.cc:302] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbde26b7bf7e4a4b9d1eeeeaa458ea34, f0142bd70c61423082fa7eceb92b5fbc; no voters: 34231b5a95474f3ebf353786e5000770
I20260925 00:53:47.253196 23901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0142bd70c61423082fa7eceb92b5fbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" is_pre_election: true
I20260925 00:53:47.253377 24596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 00:53:47.253435 24596 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 34231b5a95474f3ebf353786e5000770)
I20260925 00:53:47.253490 24596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:53:47.254477 24596 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } }
I20260925 00:53:47.254739 23901 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0142bd70c61423082fa7eceb92b5fbc in current term 2: Already voted for candidate bbde26b7bf7e4a4b9d1eeeeaa458ea34 in this term.
I20260925 00:53:47.254820 24596 leader_election.cc:288] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [CANDIDATE]: Term 2 election: Requested vote from peers 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099), f0142bd70c61423082fa7eceb92b5fbc (127.16.202.60:37597)
I20260925 00:53:47.255213 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.254989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47218)
0925 00:53:47.255096 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.255146 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 00:53:47.255300 24062 leader_election.cc:302] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0142bd70c61423082fa7eceb92b5fbc; no voters: 34231b5a95474f3ebf353786e5000770, bbde26b7bf7e4a4b9d1eeeeaa458ea34
I20260925 00:53:47.255342 24085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0142bd70c61423082fa7eceb92b5fbc"
I20260925 00:53:47.255443 24085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:53:47.255824 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.255625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47230)
0925 00:53:47.255724 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.255771 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 00:53:47.255920 23878 leader_election.cc:334] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099): Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
I20260925 00:53:47.256498 24085 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbde26b7bf7e4a4b9d1eeeeaa458ea34 in term 2.
I20260925 00:53:47.256613 24599 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:53:47.256791 23878 leader_election.cc:302] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbde26b7bf7e4a4b9d1eeeeaa458ea34, f0142bd70c61423082fa7eceb92b5fbc; no voters: 34231b5a95474f3ebf353786e5000770
I20260925 00:53:47.256963 24596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:53:47.257166 24596 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [term 2 LEADER]: Becoming Leader. State: Replica: bbde26b7bf7e4a4b9d1eeeeaa458ea34, State: Running, Role: LEADER
I20260925 00:53:47.257429 24596 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } }
I20260925 00:53:47.258386 24606 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbde26b7bf7e4a4b9d1eeeeaa458ea34. Latest consensus state: current_term: 2 leader_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:47.258481 24606 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:47.258728 24608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:47.260985 24608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:47.261386 24608 catalog_manager.cc:1259] Loaded cluster ID: cc3ac59429fa4eada86d966bc6f45633
I20260925 00:53:47.261433 24608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:47.262001 24608 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:47.262354 24608 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Loaded TSK: 0
I20260925 00:53:47.263074 24608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:53:47.305747 23891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:47.308928 23891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:47.312001 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.311740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47248)
0925 00:53:47.311848 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.311916 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 00:53:47.312029 24085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [term 2 FOLLOWER]: Refusing update from remote peer bbde26b7bf7e4a4b9d1eeeeaa458ea34: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 00:53:47.312498 24606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 00:53:47.312811 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.312572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.0.0.1:47262)
0925 00:53:47.312693 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.312747 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 00:53:47.313158 23878 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 -> Peer 34231b5a95474f3ebf353786e5000770 (127.16.202.62:45099): Couldn't send request to peer 34231b5a95474f3ebf353786e5000770. Status: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 00:53:47.314193 24599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbde26b7bf7e4a4b9d1eeeeaa458ea34. Latest consensus state: current_term: 2 leader_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:47.314313 24599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:47.315436 24606 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:47.315557 24606 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34 [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:47.315961 24610 mvcc.cc:205] Tried to move back new op lower bound from 7333059081464979456 to 7333059081248673792. Current Snapshot: MvccSnapshot[applied={T|T < 7333059081464979456}]
I20260925 00:53:47.316103 24599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34231b5a95474f3ebf353786e5000770" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45099 } } peers { permanent_uuid: "bbde26b7bf7e4a4b9d1eeeeaa458ea34" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 42451 } attrs { promote: false } } peers { permanent_uuid: "f0142bd70c61423082fa7eceb92b5fbc" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37597 } attrs { promote: false } } }
I20260925 00:53:47.316212 24599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0142bd70c61423082fa7eceb92b5fbc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:53:47.325650 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.321305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:39921 (local address 127.0.0.1:33428)
0925 00:53:47.321413 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.321429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.321505 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.322500 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.322503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.322525 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.322622 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.322628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.323611 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.323615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.324453 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.324462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.325320 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.325324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.325414 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.325443 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.325489 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.325543 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260925 00:53:47.326241 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.321762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.16.202.2:39921)
0925 00:53:47.322271 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.322276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.322289 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:47.322348 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.322351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.322397 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.322482 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.322722 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.323504 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.324592 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.325201 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.325936 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.326004 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.326041 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.326111 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":91,"threads_started":1}
I20260925 00:53:47.326479 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.321588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43423 (local address 127.0.0.1:52692)
0925 00:53:47.321983 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.321996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.322077 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.322865 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.322868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.322882 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.322976 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.322981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.323858 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.323862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.325182 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.325192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.326086 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.326089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.326217 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.326234 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.326280 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.326356 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":41}
I20260925 00:53:47.326493 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.322274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.16.202.3:44973)
0925 00:53:47.322628 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.322633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.322743 (+ 110us) server_negotiation.cc:407] Connection header received
0925 00:53:47.322866 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.322869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.322917 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.323001 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.323201 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.323837 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.325326 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.325954 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.326060 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.326122 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.326301 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.326380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":88,"threads_started":1}
I20260925 00:53:47.326732 24325 tablet_service.cc:1467] Processing CreateTablet for tablet a94e3f160fa94a0ebd5703ab6a40a541 (DEFAULT_TABLE table=test-table [id=cebaf8a20283421da316046a3deb71fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:47.326908 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.321663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.16.202.1:43423)
0925 00:53:47.322639 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.322644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.322658 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:47.322709 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.322712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.322758 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.323079 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.323192 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.323751 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.325332 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.325959 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.326660 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.326723 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.326754 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.326812 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":78,"threads_started":1}
I20260925 00:53:47.327036 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94e3f160fa94a0ebd5703ab6a40a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:47.327170 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.321799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:44973 (local address 127.0.0.1:34274)
0925 00:53:47.322672 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.322685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.322769 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.323016 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.323018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.323033 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.323108 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.323114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.323949 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.323952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.325183 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.325192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.326064 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.326067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.326180 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.326202 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.326275 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.327052 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":77,"thread_start_us":73,"threads_started":1}
I20260925 00:53:47.327382 24188 tablet_service.cc:1467] Processing CreateTablet for tablet a94e3f160fa94a0ebd5703ab6a40a541 (DEFAULT_TABLE table=test-table [id=cebaf8a20283421da316046a3deb71fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:47.327626 24188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94e3f160fa94a0ebd5703ab6a40a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:47.329219 24462 tablet_service.cc:1467] Processing CreateTablet for tablet a94e3f160fa94a0ebd5703ab6a40a541 (DEFAULT_TABLE table=test-table [id=cebaf8a20283421da316046a3deb71fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:47.329489 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94e3f160fa94a0ebd5703ab6a40a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:47.329584 24619 tablet_bootstrap.cc:492] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Bootstrap starting.
I20260925 00:53:47.330523 24619 tablet_bootstrap.cc:654] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:47.331862 24619 tablet_bootstrap.cc:492] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: No bootstrap required, opened a new log
I20260925 00:53:47.331967 24619 ts_tablet_manager.cc:1397] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:47.332134 24622 tablet_bootstrap.cc:492] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Bootstrap starting.
I20260925 00:53:47.332427 24619 raft_consensus.cc:348] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.332556 24619 raft_consensus.cc:374] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:47.332594 24619 raft_consensus.cc:729] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1795c71d5741c68314cbac9a9efea4, State: Initialized, Role: FOLLOWER
I20260925 00:53:47.332764 24619 consensus_queue.cc:260] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.333113 24619 ts_tablet_manager.cc:1428] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:53:47.333364 24622 tablet_bootstrap.cc:654] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:47.333456 24620 tablet_bootstrap.cc:492] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4: Bootstrap starting.
I20260925 00:53:47.334393 24620 tablet_bootstrap.cc:654] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:47.336666 24393 heartbeater.cc:503] Master 127.16.202.61:42451 was elected leader, sending a full tablet report...
I20260925 00:53:47.337301 24620 tablet_bootstrap.cc:492] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4: No bootstrap required, opened a new log
I20260925 00:53:47.337375 24620 ts_tablet_manager.cc:1397] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:53:47.337736 24620 raft_consensus.cc:348] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.337823 24620 raft_consensus.cc:374] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:47.337852 24620 raft_consensus.cc:729] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c33483c2e14ceb81cdb603d4dcbce4, State: Initialized, Role: FOLLOWER
I20260925 00:53:47.337976 24620 consensus_queue.cc:260] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.338325 24620 ts_tablet_manager.cc:1428] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:47.336833 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.334622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:38769)
0925 00:53:47.336590 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.336653 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1896,"thread_start_us":90,"threads_started":1}
W20260925 00:53:47.337949 24394 tablet.cc:2369] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:47.339761 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.339357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:56453)
0925 00:53:47.339603 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.339666 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260925 00:53:47.340327 24256 heartbeater.cc:503] Master 127.16.202.61:42451 was elected leader, sending a full tablet report...
I20260925 00:53:47.340708 24622 tablet_bootstrap.cc:492] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: No bootstrap required, opened a new log
I20260925 00:53:47.340891 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.338604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:33323)
0925 00:53:47.340709 (+ 2105us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.340774 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2017,"thread_start_us":94,"threads_started":1}
I20260925 00:53:47.340818 24622 ts_tablet_manager.cc:1397] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 00:53:47.341530 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.341301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:41283)
0925 00:53:47.341415 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.341469 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 00:53:47.341892 24622 raft_consensus.cc:348] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.342003 24622 raft_consensus.cc:374] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:47.342023 24622 raft_consensus.cc:729] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0719fa59798645649e691f3a6c185032, State: Initialized, Role: FOLLOWER
I20260925 00:53:47.342147 24622 consensus_queue.cc:260] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.342469 24622 ts_tablet_manager.cc:1428] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:47.343261 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.342899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:49323)
0925 00:53:47.343114 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.343171 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260925 00:53:47.343914 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.343689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:57695)
0925 00:53:47.343786 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.343836 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 00:53:47.344343 24530 heartbeater.cc:503] Master 127.16.202.61:42451 was elected leader, sending a full tablet report...
I20260925 00:53:47.380024 24627 raft_consensus.cc:482] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:47.380172 24627 raft_consensus.cc:504] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.381276 24627 leader_election.cc:288] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921), 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973)
I20260925 00:53:47.385746 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.381646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:41821 (local address 127.16.202.2:39921)
0925 00:53:47.381786 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.381791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.381808 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:53:47.381962 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.381965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.382014 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.382084 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.382412 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.383144 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.384641 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.385197 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.385306 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.385361 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.385555 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.385628 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:47.385751 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.381545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:39921 (local address 127.16.202.1:41821)
0925 00:53:47.381692 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.381708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.381849 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.382112 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.382116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.382142 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.382280 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.382287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.383252 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.383254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.384516 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.384527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.385306 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.385317 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.385441 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.385463 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.385528 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.385602 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 00:53:47.385814 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.381944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34071 (local address 127.16.202.3:44973)
0925 00:53:47.382076 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.382080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.382232 (+ 152us) server_negotiation.cc:407] Connection header received
0925 00:53:47.382343 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.382346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.382393 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.382461 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:47.382715 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.383386 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.384741 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.385282 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.385394 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.385442 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.385623 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.385696 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:53:47.386072 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.381826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:44973 (local address 127.16.202.1:34071)
0925 00:53:47.382147 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.382159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:47.382254 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:47.382509 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:47.382512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:47.382527 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:47.382617 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.382621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.383512 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.383515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.384579 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:47.384586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.385402 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.385406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:47.385516 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.385534 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:53:47.385596 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 00:53:47.385664 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":139,"threads_started":1}
I20260925 00:53:47.386318 24345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94e3f160fa94a0ebd5703ab6a40a541" candidate_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1795c71d5741c68314cbac9a9efea4" is_pre_election: true
I20260925 00:53:47.386477 24345 raft_consensus.cc:2454] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c33483c2e14ceb81cdb603d4dcbce4 in term 0.
I20260925 00:53:47.386580 24482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94e3f160fa94a0ebd5703ab6a40a541" candidate_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0719fa59798645649e691f3a6c185032" is_pre_election: true
I20260925 00:53:47.386831 24482 raft_consensus.cc:2454] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c33483c2e14ceb81cdb603d4dcbce4 in term 0.
I20260925 00:53:47.386833 24144 leader_election.cc:302] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c33483c2e14ceb81cdb603d4dcbce4, ee1795c71d5741c68314cbac9a9efea4; no voters:
I20260925 00:53:47.387086 24627 raft_consensus.cc:2781] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:47.387148 24627 raft_consensus.cc:482] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:47.387182 24627 raft_consensus.cc:3037] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:47.388173 24627 raft_consensus.cc:504] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.388492 24627 leader_election.cc:288] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [CANDIDATE]: Term 1 election: Requested vote from peers ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921), 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973)
I20260925 00:53:47.388814 24345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94e3f160fa94a0ebd5703ab6a40a541" candidate_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1795c71d5741c68314cbac9a9efea4"
I20260925 00:53:47.388875 24482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94e3f160fa94a0ebd5703ab6a40a541" candidate_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0719fa59798645649e691f3a6c185032"
I20260925 00:53:47.388926 24345 raft_consensus.cc:3037] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:47.388993 24482 raft_consensus.cc:3037] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:47.389793 24345 raft_consensus.cc:2454] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c33483c2e14ceb81cdb603d4dcbce4 in term 1.
I20260925 00:53:47.389859 24482 raft_consensus.cc:2454] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c33483c2e14ceb81cdb603d4dcbce4 in term 1.
I20260925 00:53:47.390134 24144 leader_election.cc:302] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0719fa59798645649e691f3a6c185032, 62c33483c2e14ceb81cdb603d4dcbce4; no voters:
I20260925 00:53:47.390288 24627 raft_consensus.cc:2781] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:47.390519 24627 raft_consensus.cc:686] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [term 1 LEADER]: Becoming Leader. State: Replica: 62c33483c2e14ceb81cdb603d4dcbce4, State: Running, Role: LEADER
I20260925 00:53:47.390771 24627 consensus_queue.cc:237] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } }
I20260925 00:53:47.391258 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.390975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:58079)
0925 00:53:47.391132 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.391186 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 00:53:47.393133 23891 catalog_manager.cc:5495] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c33483c2e14ceb81cdb603d4dcbce4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:47.408463 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.404911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.16.202.1:43423)
0925 00:53:47.405051 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:47.405056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:47.405070 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:47.405219 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:47.405223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:47.405278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:47.405371 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:47.405630 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.406150 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.406987 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:47.407271 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:47.407360 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:47.407621 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:47.408108 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:47.408207 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:47.408275 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 00:53:47.408330 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:53:47.411134 24345 raft_consensus.cc:1261] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Refusing update from remote peer 62c33483c2e14ceb81cdb603d4dcbce4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:47.411403 24482 raft_consensus.cc:1261] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Refusing update from remote peer 62c33483c2e14ceb81cdb603d4dcbce4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:47.411696 24633 consensus_queue.cc:1035] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1795c71d5741c68314cbac9a9efea4" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:47.411969 24627 consensus_queue.cc:1035] T a94e3f160fa94a0ebd5703ab6a40a541 P 62c33483c2e14ceb81cdb603d4dcbce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0719fa59798645649e691f3a6c185032" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:47.413923 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.413365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:49089)
0925 00:53:47.413786 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.413848 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
I20260925 00:53:47.414677 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.414403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:56151)
0925 00:53:47.414549 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.414608 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 00:53:47.417425 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.417200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:59661)
0925 00:53:47.417292 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.417352 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 00:53:47.418900 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.418543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.3:45479)
0925 00:53:47.418780 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.418832 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 00:53:47.419047 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.418839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.1:47523)
0925 00:53:47.418939 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.418986 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 00:53:47.419242 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.418868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45099 (local address 127.16.202.2:51517)
0925 00:53:47.419124 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.419176 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260925 00:53:47.426340 23890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:47.426506 23890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:47.430171 23890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbde26b7bf7e4a4b9d1eeeeaa458ea34: Sending DeleteTablet for 3 replicas of tablet a94e3f160fa94a0ebd5703ab6a40a541
I20260925 00:53:47.430766 24325 tablet_service.cc:1514] Processing DeleteTablet for tablet a94e3f160fa94a0ebd5703ab6a40a541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:47 UTC) from {username='oryx', principal='oryx/127.16.202.61@KRBTEST.COM'} at 127.0.0.1:33428
I20260925 00:53:47.431164 24647 tablet_replica.cc:330] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: stopping tablet replica
I20260925 00:53:47.431324 24647 raft_consensus.cc:2229] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:47.431329 24462 tablet_service.cc:1514] Processing DeleteTablet for tablet a94e3f160fa94a0ebd5703ab6a40a541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:47 UTC) from {username='oryx', principal='oryx/127.16.202.61@KRBTEST.COM'} at 127.0.0.1:34274
I20260925 00:53:47.431465 24647 raft_consensus.cc:2258] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:47.431862 24188 tablet_service.cc:1514] Processing DeleteTablet for tablet a94e3f160fa94a0ebd5703ab6a40a541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:47 UTC) from {username='oryx', principal='oryx/127.16.202.61@KRBTEST.COM'} at 127.0.0.1:52692
I20260925 00:53:47.432199 24648 tablet_replica.cc:330] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: stopping tablet replica
I20260925 00:53:47.432221 24647 ts_tablet_manager.cc:1905] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:47.432315 24648 raft_consensus.cc:2229] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:47.432432 24648 raft_consensus.cc:2258] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:47.433162 24648 ts_tablet_manager.cc:1905] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:47.433815 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 24035
I20260925 00:53:47.435009 24647 ts_tablet_manager.cc:1918] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:47.435081 24647 log.cc:1196] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/a94e3f160fa94a0ebd5703ab6a40a541
I20260925 00:53:47.435369 24647 ts_tablet_manager.cc:1939] T a94e3f160fa94a0ebd5703ab6a40a541 P ee1795c71d5741c68314cbac9a9efea4: Deleting consensus metadata
I20260925 00:53:47.435734 24648 ts_tablet_manager.cc:1918] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:47.435806 24648 log.cc:1196] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/a94e3f160fa94a0ebd5703ab6a40a541
I20260925 00:53:47.436062 24648 ts_tablet_manager.cc:1939] T a94e3f160fa94a0ebd5703ab6a40a541 P 0719fa59798645649e691f3a6c185032: Deleting consensus metadata
I20260925 00:53:47.436369 23878 catalog_manager.cc:4841] TS ee1795c71d5741c68314cbac9a9efea4 (127.16.202.2:39921): tablet a94e3f160fa94a0ebd5703ab6a40a541 (table test-table [id=cebaf8a20283421da316046a3deb71fa]) successfully deleted
I20260925 00:53:47.437275 23878 catalog_manager.cc:4841] TS 0719fa59798645649e691f3a6c185032 (127.16.202.3:44973): tablet a94e3f160fa94a0ebd5703ab6a40a541 (table test-table [id=cebaf8a20283421da316046a3deb71fa]) successfully deleted
I20260925 00:53:47.440353 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 24262
I20260925 00:53:47.441074 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:47.440553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43423 (local address 127.0.0.1:52716)
0925 00:53:47.440943 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:47.440995 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
W20260925 00:53:47.441267 23877 catalog_manager.cc:4584] TS 62c33483c2e14ceb81cdb603d4dcbce4 (127.16.202.1:43423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a94e3f160fa94a0ebd5703ab6a40a541: Network error: Client connection negotiation failed: client connection to 127.16.202.1:43423: connect: Connection refused (error 111)
I20260925 00:53:47.445749 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 24398
I20260925 00:53:47.451200 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23755
I20260925 00:53:47.456097 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 23855
2026-09-25T00:53:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5249 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:47 dist-test-slave-rpk1 krb5kdc[24652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:47 dist-test-slave-rpk1 krb5kdc[24652](info): set up 2 sockets
Sep 25 00:53:47 dist-test-slave-rpk1 krb5kdc[24652](info): commencing operation
krb5kdc: starting...
W20260925 00:53:48.421089 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:48 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab.
Entry for principal oryx/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab.
I20260925 00:53:48.541443 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36985
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab
--principal=oryx/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:48.654589 24665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:48.654894 24665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:48.654953 24665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:48.659209 24665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:48.659300 24665 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 00:53:48.659332 24665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:48.659354 24665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:48.659377 24665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:48.659401 24665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:48.665138 24665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36985
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.62.keytab
--principal=oryx/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:48.666334 24665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:48.667459 24665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:48.676205 24671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:48.676221 24670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:48.676425 24665 server_base.cc:1034] running on GCE node
W20260925 00:53:48.676205 24673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:48.677124 24665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:48.678258 24665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:48.679414 24665 hybrid_clock.cc:648] HybridClock initialized: now 1790297628679395 us; error 32 us; skew 500 ppm
Sep 25 00:53:48 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:48.682158 24665 init.cc:377] Logged in from keytab as oryx/127.16.202.62@KRBTEST.COM (short username oryx)
I20260925 00:53:48.683065 24665 webserver.cc:460] Webserver started at http://127.16.202.62:38039/ using document root <none> and password file <none>
I20260925 00:53:48.683642 24665 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:48.683708 24665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:48.683899 24665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:48.686134 24665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "cb7cc9af475d44ccbf1c3428642f2dcc"
format_stamp: "Formatted at 2026-09-25 00:53:48 on dist-test-slave-rpk1"
server_key: "0f7d47a121f2d6adfbe38f63fb5378e5"
server_key_iv: "2fb185fe2e04c26367f7941242e308ea"
server_key_version: "encryptionkey@0"
I20260925 00:53:48.686679 24665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "cb7cc9af475d44ccbf1c3428642f2dcc"
format_stamp: "Formatted at 2026-09-25 00:53:48 on dist-test-slave-rpk1"
server_key: "0f7d47a121f2d6adfbe38f63fb5378e5"
server_key_iv: "2fb185fe2e04c26367f7941242e308ea"
server_key_version: "encryptionkey@0"
I20260925 00:53:48.690569 24665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:48.693013 24680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:48.694258 24665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:48.694392 24665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "cb7cc9af475d44ccbf1c3428642f2dcc"
format_stamp: "Formatted at 2026-09-25 00:53:48 on dist-test-slave-rpk1"
server_key: "0f7d47a121f2d6adfbe38f63fb5378e5"
server_key_iv: "2fb185fe2e04c26367f7941242e308ea"
server_key_version: "encryptionkey@0"
I20260925 00:53:48.694495 24665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:48.708907 24665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:48.711915 24665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:48.712105 24665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:48.719754 24665 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36985
I20260925 00:53:48.719784 24742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36985 every 8 connection(s)
I20260925 00:53:48.720715 24665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:48.723430 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:48.728063 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 24665
I20260925 00:53:48.728192 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:48.728503 17192 external_mini_cluster.cc:1428] Setting key 25576d8b0bd8fc87d1c9a549d17952cf
I20260925 00:53:48.729528 24743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Bootstrap starting.
I20260925 00:53:48.731988 24743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:48.732983 24743 log.cc:824] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:48.735342 24743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: No bootstrap required, opened a new log
Sep 25 00:53:48 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:48.739660 24743 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36985 } }
I20260925 00:53:48.739918 24743 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:48.739975 24743 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7cc9af475d44ccbf1c3428642f2dcc, State: Initialized, Role: FOLLOWER
I20260925 00:53:48.740460 24743 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36985 } }
I20260925 00:53:48.740653 24743 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:48.740749 24743 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:48.740878 24743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:48.741827 24743 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36985 } }
I20260925 00:53:48.742159 24743 leader_election.cc:302] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb7cc9af475d44ccbf1c3428642f2dcc; no voters:
I20260925 00:53:48.742460 24743 leader_election.cc:288] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:48.742615 24748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:48.742791 24748 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [term 1 LEADER]: Becoming Leader. State: Replica: cb7cc9af475d44ccbf1c3428642f2dcc, State: Running, Role: LEADER
I20260925 00:53:48.743081 24748 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36985 } }
I20260925 00:53:48.743760 24743 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:48.744001 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:48.730028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.16.202.62:36985)
0925 00:53:48.730458 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:48.730475 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:48.730512 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:53:48.731191 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:48.731209 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:48.731568 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:48.731873 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:48.732799 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:48.734050 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:48.735230 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:48.735748 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:48.735868 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:48.738625 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:48.738630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:48.738646 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:48.738675 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:48.740274 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:48.740843 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:48.740849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:48.740856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:48.741020 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:48.741308 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:48.741311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:48.741313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:48.741746 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:48.741924 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:48.742241 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 00:53:48.742579 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":116,"threads_started":1}
I20260925 00:53:48.744231 24748 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb7cc9af475d44ccbf1c3428642f2dcc. Latest consensus state: current_term: 1 leader_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36985 } } }
I20260925 00:53:48.744343 24748 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:48.745764 24749 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7cc9af475d44ccbf1c3428642f2dcc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36985 } } }
I20260925 00:53:48.746476 24749 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:48.747097 24752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:48.750016 24752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:48.755241 24752 catalog_manager.cc:1347] Generated new cluster ID: 9003e8ab656843bb9e31f602a9f6d4ba
I20260925 00:53:48.755324 24752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:48.797070 24752 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:48.798434 24752 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:48.811784 24752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Generated new TSK 0
I20260925 00:53:48.812517 24752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab.
Entry for principal oryx/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab.
I20260925 00:53:48.851239 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36985
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:48.982288 24769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:48.982590 24769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:48.982637 24769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:48.986726 24769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:48.986800 24769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:48.986847 24769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:48.986881 24769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:48.992280 24769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:36985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:48.993386 24769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:48.994356 24769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:49.003805 24774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:49.003786 24777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:49.003973 24769 server_base.cc:1034] running on GCE node
W20260925 00:53:49.003839 24775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:49.004493 24769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:49.005201 24769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:49.006357 24769 hybrid_clock.cc:648] HybridClock initialized: now 1790297629006338 us; error 31 us; skew 500 ppm
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:49.009095 24769 init.cc:377] Logged in from keytab as oryx/127.16.202.1@KRBTEST.COM (short username oryx)
I20260925 00:53:49.009989 24769 webserver.cc:460] Webserver started at http://127.16.202.1:46253/ using document root <none> and password file <none>
I20260925 00:53:49.010545 24769 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:49.010614 24769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:49.010838 24769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:49.012987 24769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "1aa9682f65a44e719f458211704e1a8e"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "e5140b945bf29b3d07a53d55690cc69c"
server_key_iv: "e2f6e0763588585bb4f5e2e1ed2391bb"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.013541 24769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "1aa9682f65a44e719f458211704e1a8e"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "e5140b945bf29b3d07a53d55690cc69c"
server_key_iv: "e2f6e0763588585bb4f5e2e1ed2391bb"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.017346 24769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:53:49.019824 24784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.021055 24769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:53:49.021162 24769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "1aa9682f65a44e719f458211704e1a8e"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "e5140b945bf29b3d07a53d55690cc69c"
server_key_iv: "e2f6e0763588585bb4f5e2e1ed2391bb"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.021256 24769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:49.038755 24769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:49.041988 24769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:49.042155 24769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:49.043627 24769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:49.043689 24769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.043742 24769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:49.043763 24769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:49.054023 24769 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:39565
I20260925 00:53:49.054075 24899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:39565 every 8 connection(s)
I20260925 00:53:49.055084 24769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:49.057495 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.044612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:56943 (local address 127.16.202.62:36985)
0925 00:53:49.044756 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.044760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.045970 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 00:53:49.047132 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.047137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.047202 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.047303 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.048793 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.049323 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.050462 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.050697 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.050790 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.053974 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.053978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.053981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.054014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.055404 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.055956 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.055959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.055960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.056041 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.056358 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.056360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.056362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.056548 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.056633 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.057176 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.057333 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:53:49.058221 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.045296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.16.202.1:56943)
0925 00:53:49.045819 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.045878 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.046867 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.047484 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.047494 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.047897 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:49.048619 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.048639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.049446 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.049449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.050343 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.050354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.050855 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.050860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.051037 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.052250 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:49.052269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:49.053787 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:49.055512 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.055518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.055534 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.055852 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.056124 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.056127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.056129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.056258 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.056647 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:49.056654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:49.056938 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.057097 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.057407 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260925 00:53:49.058688 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 24769
I20260925 00:53:49.058825 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:49.059170 17192 external_mini_cluster.cc:1428] Setting key cf3e21be71d8b1172d8f177f4326ecb6
I20260925 00:53:49.059717 24900 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36985
I20260925 00:53:49.059861 24900 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:49.060352 24900 heartbeater.cc:511] Master 127.16.202.62:36985 requested a full tablet report, sending...
I20260925 00:53:49.061956 24696 ts_manager.cc:194] Registered new tserver with Master: 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565)
I20260925 00:53:49.063340 24696 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:56943
WARNING: no policy specified for oryx/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:53:49.069591 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.061917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.16.202.1:38669)
0925 00:53:49.062225 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.062239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.062338 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.062653 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.062656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.062883 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:49.063002 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.063008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.064021 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.064025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.064894 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.064905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.065489 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.065493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.065593 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.066081 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:49.066096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:49.066548 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:49.068109 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.068113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.068116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.068369 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.068644 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.068647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.068649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.068764 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.069139 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:49.069142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:49.069232 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.069323 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.069435 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260925 00:53:49.069602 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.061978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:38669 (local address 127.16.202.62:36985)
0925 00:53:49.062089 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.062093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.062313 (+ 220us) server_negotiation.cc:407] Connection header received
0925 00:53:49.062420 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.062423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.062465 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.062540 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.063124 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.063846 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.065049 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.065356 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.065483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.066686 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.066689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.066691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.066714 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.067998 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.068490 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.068492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.068494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.068565 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.068846 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.068849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.068851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.069048 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.069124 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.069357 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.069483 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab.
Entry for principal oryx/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab.
I20260925 00:53:49.087385 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36985
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:49.198931 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:49.199211 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:49.199258 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:49.203450 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:49.203511 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:49.203552 24909 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:49.203590 24909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:49.208933 24909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:36985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:49.209959 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:49.210966 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:49.220474 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:49.220619 24909 server_base.cc:1034] running on GCE node
W20260925 00:53:49.220489 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:49.220521 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:49.221133 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:49.221804 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:49.223021 24909 hybrid_clock.cc:648] HybridClock initialized: now 1790297629222999 us; error 35 us; skew 500 ppm
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:49.225989 24909 init.cc:377] Logged in from keytab as oryx/127.16.202.2@KRBTEST.COM (short username oryx)
I20260925 00:53:49.226892 24909 webserver.cc:460] Webserver started at http://127.16.202.2:43657/ using document root <none> and password file <none>
I20260925 00:53:49.227443 24909 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:49.227505 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:49.227689 24909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:49.229786 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "1e2654fbd8a247499d319f44d8adab79"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "79d472cfeb8f22d8a53dbecab97f0217"
server_key_iv: "8ed01ac1503831ff181fbec412009ff6"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.230300 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "1e2654fbd8a247499d319f44d8adab79"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "79d472cfeb8f22d8a53dbecab97f0217"
server_key_iv: "8ed01ac1503831ff181fbec412009ff6"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.234151 24909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:49.236863 24924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.238548 24909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:49.238713 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "1e2654fbd8a247499d319f44d8adab79"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "79d472cfeb8f22d8a53dbecab97f0217"
server_key_iv: "8ed01ac1503831ff181fbec412009ff6"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.239110 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:49.256431 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:49.259958 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:49.260151 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:49.261924 24909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:49.261986 24909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.262040 24909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:49.262073 24909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.271763 24909 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:37429
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:49.272895 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:49.272898 25039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:37429 every 8 connection(s)
I20260925 00:53:49.274266 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 24909
I20260925 00:53:49.274394 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:49.274718 17192 external_mini_cluster.cc:1428] Setting key 53fe58e5c1a508f28f1794e09355283d
I20260925 00:53:49.276976 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.262751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:52935 (local address 127.16.202.62:36985)
0925 00:53:49.262903 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.262908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.263828 (+ 920us) server_negotiation.cc:407] Connection header received
0925 00:53:49.265021 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.265026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.265084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.265173 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.266872 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.267685 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.269414 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.269631 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.269736 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.272885 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.272890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.272893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.272927 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.274663 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.275242 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.275245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.275246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.275337 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.275729 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.275733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.275734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.275922 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.276015 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.276715 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.276842 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:53:49.278026 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.263178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.16.202.2:52935)
0925 00:53:49.263649 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.263701 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.264728 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.265441 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.265452 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.265965 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:49.266713 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.266736 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.267822 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.267826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.269285 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.269297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.269797 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.269802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.270031 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.270728 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:49.270750 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:49.272648 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:49.274796 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.274802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.274819 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.275105 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.275436 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.275439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.275441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.275626 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.276033 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:49.276040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:49.276360 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.276580 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.276961 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260925 00:53:49.279393 25040 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36985
I20260925 00:53:49.279546 25040 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:49.280071 25040 heartbeater.cc:511] Master 127.16.202.62:36985 requested a full tablet report, sending...
I20260925 00:53:49.281141 24697 ts_manager.cc:194] Registered new tserver with Master: 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429)
I20260925 00:53:49.281818 24697 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:52935
WARNING: no policy specified for oryx/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:49.290215 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.281852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:49667 (local address 127.16.202.62:36985)
0925 00:53:49.281959 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.281963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.282294 (+ 331us) server_negotiation.cc:407] Connection header received
0925 00:53:49.282399 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.282403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.282454 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.282525 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.284060 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.284876 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.285728 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.285944 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.286026 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.287073 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.287076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.287078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.287101 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.288570 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.289067 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.289071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.289073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.289158 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.289430 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.289433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.289435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.289646 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.289723 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.290000 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.290092 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 00:53:49.290302 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.281746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.16.202.2:49667)
0925 00:53:49.282208 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.282222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.282320 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.283642 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.283645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.283842 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:49.283936 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.283941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.285021 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.285024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.285619 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.285628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.286071 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.286075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.286158 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.286591 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:49.286602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:49.286900 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:49.288744 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.288748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.288749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.288968 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.289241 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.289244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.289246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.289340 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.289728 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:49.289731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:49.289825 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.289959 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.290106 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab.
Entry for principal oryx/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab.
I20260925 00:53:49.304198 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36985
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:49.417649 25049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:49.417941 25049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:49.417994 25049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:49.422076 25049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:49.422133 25049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:49.422180 25049 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:53:49.422237 25049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:49.427763 25049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.202.62:36985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:49.428941 25049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:49.430014 25049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:49.440521 25055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:49.440759 25049 server_base.cc:1034] running on GCE node
W20260925 00:53:49.440514 25057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:49.440546 25054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:49.441277 25049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:49.441960 25049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:49.443125 25049 hybrid_clock.cc:648] HybridClock initialized: now 1790297629443104 us; error 40 us; skew 500 ppm
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:49.446024 25049 init.cc:377] Logged in from keytab as oryx/127.16.202.3@KRBTEST.COM (short username oryx)
I20260925 00:53:49.446991 25049 webserver.cc:460] Webserver started at http://127.16.202.3:34885/ using document root <none> and password file <none>
I20260925 00:53:49.447553 25049 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:49.447616 25049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:49.447814 25049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:49.449903 25049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "723536ea12274114b2035fa62ef99adf"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "0545fbbd84219aac89adc8f7419c2b02"
server_key_iv: "e88740272ed8f13e0ab0f68c941aca52"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.450445 25049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "723536ea12274114b2035fa62ef99adf"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "0545fbbd84219aac89adc8f7419c2b02"
server_key_iv: "e88740272ed8f13e0ab0f68c941aca52"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.454675 25049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:49.457440 25064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.458917 25049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:49.459041 25049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "723536ea12274114b2035fa62ef99adf"
format_stamp: "Formatted at 2026-09-25 00:53:49 on dist-test-slave-rpk1"
server_key: "0545fbbd84219aac89adc8f7419c2b02"
server_key_iv: "e88740272ed8f13e0ab0f68c941aca52"
server_key_version: "encryptionkey@0"
I20260925 00:53:49.459169 25049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:49.471333 25049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:49.474436 25049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:49.474620 25049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:49.476111 25049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:49.476163 25049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:49.476217 25049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:49.476246 25049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:49.486672 25049 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:41705
I20260925 00:53:49.486807 25179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:41705 every 8 connection(s)
I20260925 00:53:49.487612 25049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:49.490583 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.477135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:60455 (local address 127.16.202.62:36985)
0925 00:53:49.477264 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.477267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.478359 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 00:53:49.479413 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.479417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.479489 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.479564 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.481381 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.482224 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.484030 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.484280 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.484374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.487336 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.487340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.487342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.487370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.488866 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.489341 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.489343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.489344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.489419 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.489689 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.489691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.489692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.489872 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.489941 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.490373 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.490465 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:53:49.490614 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25049
I20260925 00:53:49.490731 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:49.491029 17192 external_mini_cluster.cc:1428] Setting key 2f6fd197ae0bb086a387e2dd6bb60128
I20260925 00:53:49.491704 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.477699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.16.202.3:60455)
0925 00:53:49.478204 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.478249 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.479181 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.479860 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.479871 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.480377 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:49.481188 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.481212 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.482402 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.482406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.483882 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.483894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.484447 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.484452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.484703 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.485444 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:49.485466 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:49.487149 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:49.488979 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.488985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.488999 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.489249 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.489492 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.489494 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.489495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.489622 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.489941 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:49.489946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:49.490189 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.490348 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.490677 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260925 00:53:49.492959 25180 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36985
I20260925 00:53:49.493108 25180 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:49.493673 25180 heartbeater.cc:511] Master 127.16.202.62:36985 requested a full tablet report, sending...
I20260925 00:53:49.494760 24696 ts_manager.cc:194] Registered new tserver with Master: 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:49.495419 24696 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:60455
I20260925 00:53:49.502177 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.494979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:48855 (local address 127.16.202.62:36985)
0925 00:53:49.495727 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.495732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.495746 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:49.495783 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.495786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.495824 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.495889 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.496357 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.497130 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.497915 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.498129 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.498230 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.499277 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.499281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.499282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.499302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.500494 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.500971 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.500975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.500976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.501066 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.501344 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.501348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.501350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.501546 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.501641 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.501904 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.502015 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671}
I20260925 00:53:49.502197 25186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.494898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.16.202.3:48855)
0925 00:53:49.495158 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.495170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.495238 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.495922 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.495925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.496106 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:49.496244 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.496250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.497249 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.497252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.497813 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.497821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.498313 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.498317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.498406 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.498840 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:49.498851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:49.499156 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:49.500609 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.500613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.500615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.500877 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.501153 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:49.501157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:49.501159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:49.501252 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:49.501639 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:49.501642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:49.501745 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.501840 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.501990 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":6,"thread_start_us":113,"threads_started":1}
I20260925 00:53:49.505419 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:53:49.527980 24746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:49.521851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.16.202.62:36985)
0925 00:53:49.521984 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.521988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.522000 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:53:49.522042 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.522044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.522083 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.522151 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.522962 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.523488 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.524362 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.524580 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.524705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.527843 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55786: BlockingRecv error: recv got EOF from 127.0.0.1:55786 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 00:53:49 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:49.537667 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.529463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.16.202.62:36985)
0925 00:53:49.529577 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.529581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.529684 (+ 103us) server_negotiation.cc:407] Connection header received
0925 00:53:49.529791 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.529793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.529834 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.529904 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:49.530526 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.531040 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.531933 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.532165 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.532251 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.534925 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:49.534929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:49.534931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:49.534963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:49.536133 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.536548 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.536551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.536552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.536619 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:49.536855 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:49.536858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:49.536859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:49.537045 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:49.537148 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.537424 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.537531 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:53:49.540562 24696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:49.543682 24696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:49.558476 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.552626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39565 (local address 127.0.0.1:36022)
0925 00:53:49.553503 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.553559 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.553690 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.554459 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.554464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.554482 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:49.554662 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.554670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.555678 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.555682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.556865 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.556874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.557889 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.557893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.558007 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.558045 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.558162 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.558295 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":50,"thread_start_us":113,"threads_started":1}
I20260925 00:53:49.558494 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.552816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.16.202.1:39565)
0925 00:53:49.553510 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.553523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.553843 (+ 320us) server_negotiation.cc:407] Connection header received
0925 00:53:49.553928 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.553933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.554157 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.554309 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:49.554786 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.555566 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.556991 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.557550 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.557646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.557823 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.558228 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.558341 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":86,"threads_started":1}
I20260925 00:53:49.558887 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.553395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37429 (local address 127.0.0.1:42518)
0925 00:53:49.554145 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.554156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.554234 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.554688 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.554691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.554720 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:49.554792 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.554798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.556128 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.556131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.557453 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.557473 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.558506 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.558509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.558591 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.558618 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.558687 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.558756 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":104,"thread_start_us":98,"threads_started":1}
I20260925 00:53:49.559257 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.552937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.16.202.3:41705)
0925 00:53:49.553334 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.553347 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.553735 (+ 388us) server_negotiation.cc:407] Connection header received
0925 00:53:49.553974 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.553980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.554178 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.554310 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:49.554777 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.555992 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.557637 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.558436 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.558535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.558690 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.559017 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.559130 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260925 00:53:49.559346 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.553816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.16.202.2:37429)
0925 00:53:49.554193 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.554204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.554224 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:53:49.554351 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.554356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.554559 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.554697 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:49.554893 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.555992 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.557609 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.558151 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.558267 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.558427 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.559108 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.559213 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":101,"threads_started":1}
I20260925 00:53:49.559826 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.552909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41705 (local address 127.0.0.1:49812)
0925 00:53:49.553643 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.553655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.553812 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.554490 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.554493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.554512 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:49.554654 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.554662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.556128 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.556131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.557519 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.557527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.558752 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.558756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.558863 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.558886 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.559612 (+ 726us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.559695 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":27,"thread_start_us":105,"threads_started":1}
I20260925 00:53:49.562275 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0bb07588a24da8a96121cb4f027046 (DEFAULT_TABLE table=test-table [id=134e42fe4250431099d2d6cbb8f31c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:49.562261 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0bb07588a24da8a96121cb4f027046 (DEFAULT_TABLE table=test-table [id=134e42fe4250431099d2d6cbb8f31c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:49.563496 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0bb07588a24da8a96121cb4f027046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:49.563495 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0bb07588a24da8a96121cb4f027046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:49.565372 25114 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0bb07588a24da8a96121cb4f027046 (DEFAULT_TABLE table=test-table [id=134e42fe4250431099d2d6cbb8f31c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:49.566326 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0bb07588a24da8a96121cb4f027046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:49.571317 25208 tablet_bootstrap.cc:492] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Bootstrap starting.
I20260925 00:53:49.572750 25209 tablet_bootstrap.cc:492] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Bootstrap starting.
I20260925 00:53:49.573215 25210 tablet_bootstrap.cc:492] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Bootstrap starting.
I20260925 00:53:49.574186 25208 tablet_bootstrap.cc:654] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:49.575012 25209 tablet_bootstrap.cc:654] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:49.575079 25208 log.cc:824] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:49.575271 25210 tablet_bootstrap.cc:654] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:49.576051 25210 log.cc:824] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:49.576135 25209 log.cc:824] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:49.577329 25208 tablet_bootstrap.cc:492] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: No bootstrap required, opened a new log
I20260925 00:53:49.577534 25208 ts_tablet_manager.cc:1397] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:53:49.577838 25210 tablet_bootstrap.cc:492] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: No bootstrap required, opened a new log
I20260925 00:53:49.578023 25210 ts_tablet_manager.cc:1397] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:53:49.578354 25209 tablet_bootstrap.cc:492] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: No bootstrap required, opened a new log
I20260925 00:53:49.578531 25209 ts_tablet_manager.cc:1397] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:53:49.580942 25208 raft_consensus.cc:348] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.581218 25208 raft_consensus.cc:374] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:49.581262 25208 raft_consensus.cc:729] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2654fbd8a247499d319f44d8adab79, State: Initialized, Role: FOLLOWER
I20260925 00:53:49.581758 25208 consensus_queue.cc:260] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.582155 25210 raft_consensus.cc:348] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.582414 25210 raft_consensus.cc:374] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:49.582464 25210 raft_consensus.cc:729] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723536ea12274114b2035fa62ef99adf, State: Initialized, Role: FOLLOWER
I20260925 00:53:49.582445 25209 raft_consensus.cc:348] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.582674 25209 raft_consensus.cc:374] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:49.582716 25209 raft_consensus.cc:729] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Initialized, Role: FOLLOWER
I20260925 00:53:49.582857 25208 ts_tablet_manager.cc:1428] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:53:49.582911 25210 consensus_queue.cc:260] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.583061 25209 consensus_queue.cc:260] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.583240 25040 heartbeater.cc:503] Master 127.16.202.62:36985 was elected leader, sending a full tablet report...
I20260925 00:53:49.583858 25209 ts_tablet_manager.cc:1428] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:53:49.583845 25180 heartbeater.cc:503] Master 127.16.202.62:36985 was elected leader, sending a full tablet report...
I20260925 00:53:49.583858 25210 ts_tablet_manager.cc:1428] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:53:49.584909 24900 heartbeater.cc:503] Master 127.16.202.62:36985 was elected leader, sending a full tablet report...
W20260925 00:53:49.738862 25181 tablet.cc:2369] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:49.774152 25041 tablet.cc:2369] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:53:49.806942 24901 tablet.cc:2369] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:53:49.821796 25216 raft_consensus.cc:482] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:49.822011 25216 raft_consensus.cc:504] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.823094 25216 leader_election.cc:288] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705), 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429)
I20260925 00:53:49.827251 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.823934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:55497 (local address 127.16.202.2:37429)
0925 00:53:49.824078 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.824083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.824215 (+ 132us) server_negotiation.cc:407] Connection header received
0925 00:53:49.824375 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.824379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.824425 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.824489 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:49.824753 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.825246 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.826283 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.826668 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.826771 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.826824 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.827055 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.827125 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:53:49.827250 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.823541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:50831 (local address 127.16.202.3:41705)
0925 00:53:49.823688 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.823692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.823791 (+ 99us) server_negotiation.cc:407] Connection header received
0925 00:53:49.823990 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.823993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.824047 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.824124 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:49.824453 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.825126 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.826282 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.826676 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.826788 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.826837 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.827057 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.827126 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:53:49.827275 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.823808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37429 (local address 127.16.202.1:55497)
0925 00:53:49.824137 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.824147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.824233 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.824512 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.824514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.824529 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:49.824630 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.824635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.825353 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.825356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.826176 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.826185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.826771 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.826775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.826897 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.826951 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.827023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.827092 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":47,"thread_start_us":89,"threads_started":1}
I20260925 00:53:49.827283 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.823381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41705 (local address 127.16.202.1:50831)
0925 00:53:49.823695 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.823709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.823858 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.824150 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.824153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.824172 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:49.824345 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.824352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.825221 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.825225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.826171 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:49.826180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.826785 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.826788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:49.826927 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.826974 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 00:53:49.827041 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:53:49.827111 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":26,"thread_start_us":136,"threads_started":1}
I20260925 00:53:49.827854 24994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0bb07588a24da8a96121cb4f027046" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79" is_pre_election: true
I20260925 00:53:49.827942 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0bb07588a24da8a96121cb4f027046" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf" is_pre_election: true
I20260925 00:53:49.828202 24994 raft_consensus.cc:2454] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 0.
I20260925 00:53:49.828202 25134 raft_consensus.cc:2454] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 0.
I20260925 00:53:49.828773 24787 leader_election.cc:302] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:49.829044 25216 raft_consensus.cc:2781] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:49.829129 25216 raft_consensus.cc:482] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:49.829176 25216 raft_consensus.cc:3037] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:49.830197 25216 raft_consensus.cc:504] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.830513 25216 leader_election.cc:288] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 election: Requested vote from peers 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705), 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429)
I20260925 00:53:49.830899 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0bb07588a24da8a96121cb4f027046" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf"
I20260925 00:53:49.830888 24994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0bb07588a24da8a96121cb4f027046" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79"
I20260925 00:53:49.831015 25134 raft_consensus.cc:3037] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:49.831015 24994 raft_consensus.cc:3037] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:49.832304 24994 raft_consensus.cc:2454] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 1.
I20260925 00:53:49.832621 24787 leader_election.cc:302] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 1e2654fbd8a247499d319f44d8adab79; no voters:
I20260925 00:53:49.832656 25134 raft_consensus.cc:2454] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 1.
I20260925 00:53:49.832834 25216 raft_consensus.cc:2781] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:49.833060 25216 raft_consensus.cc:686] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 1 LEADER]: Becoming Leader. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Running, Role: LEADER
I20260925 00:53:49.833434 25216 consensus_queue.cc:237] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } }
I20260925 00:53:49.836642 24697 catalog_manager.cc:5495] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1aa9682f65a44e719f458211704e1a8e (127.16.202.1). New cstate: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:49.851949 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.848413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.16.202.1:39565)
0925 00:53:49.848541 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.848545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.848559 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:49.848717 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.848720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.848785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.848891 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:49.849076 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.849581 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.850472 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:49.850719 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:49.850808 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:49.851047 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:49.851529 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:49.851616 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:49.851729 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 00:53:49.851804 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:53:49.857888 24994 raft_consensus.cc:1261] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Refusing update from remote peer 1aa9682f65a44e719f458211704e1a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:49.857888 25134 raft_consensus.cc:1261] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Refusing update from remote peer 1aa9682f65a44e719f458211704e1a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:49.858548 25216 consensus_queue.cc:1035] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:49.858757 25219 consensus_queue.cc:1035] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:49.866349 25224 mvcc.cc:205] Tried to move back new op lower bound from 7333059091890495488 to 7333059091800543232. Current Snapshot: MvccSnapshot[applied={T|T < 7333059091890495488}]
I20260925 00:53:49.868266 25223 mvcc.cc:205] Tried to move back new op lower bound from 7333059091890495488 to 7333059091800543232. Current Snapshot: MvccSnapshot[applied={T|T < 7333059091890495488}]
I20260925 00:53:49.875761 24695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:49.875968 24695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:49.878844 24695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Sending DeleteTablet for 3 replicas of tablet 5b0bb07588a24da8a96121cb4f027046
I20260925 00:53:49.879675 24834 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0bb07588a24da8a96121cb4f027046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:49 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36022
I20260925 00:53:49.879863 25114 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0bb07588a24da8a96121cb4f027046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:49 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49812
I20260925 00:53:49.880025 24974 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0bb07588a24da8a96121cb4f027046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:49 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:42518
I20260925 00:53:49.880412 25232 tablet_replica.cc:330] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: stopping tablet replica
I20260925 00:53:49.880744 25231 tablet_replica.cc:330] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: stopping tablet replica
I20260925 00:53:49.880795 25232 raft_consensus.cc:2229] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:49.880831 24695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:53:49.881156 25232 raft_consensus.cc:2258] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 00:53:49.881326 24695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:49.881464 25231 raft_consensus.cc:2229] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:49.882028 25231 raft_consensus.cc:2258] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:49.882870 25232 ts_tablet_manager.cc:1905] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:49.882926 25233 tablet_replica.cc:330] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: stopping tablet replica
I20260925 00:53:49.883301 25233 raft_consensus.cc:2229] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:49.884380 25231 ts_tablet_manager.cc:1905] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:49.886178 25232 ts_tablet_manager.cc:1918] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:49.886271 25232 log.cc:1196] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/5b0bb07588a24da8a96121cb4f027046
I20260925 00:53:49.886557 25232 ts_tablet_manager.cc:1939] T 5b0bb07588a24da8a96121cb4f027046 P 723536ea12274114b2035fa62ef99adf: Deleting consensus metadata
I20260925 00:53:49.886788 25233 raft_consensus.cc:2258] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:49.887609 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 66d124de700748eaa9833cd9467eb4b8 (DEFAULT_TABLE table=test-table [id=d000c6c3d63c4df6ab97ea246702199f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:49.887873 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d124de700748eaa9833cd9467eb4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:49.887854 25114 tablet_service.cc:1467] Processing CreateTablet for tablet 66d124de700748eaa9833cd9467eb4b8 (DEFAULT_TABLE table=test-table [id=d000c6c3d63c4df6ab97ea246702199f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:49.888069 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d124de700748eaa9833cd9467eb4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:49.888540 25233 ts_tablet_manager.cc:1905] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:49.890265 25210 tablet_bootstrap.cc:492] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Bootstrap starting.
I20260925 00:53:49.890285 25208 tablet_bootstrap.cc:492] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Bootstrap starting.
I20260925 00:53:49.891181 25210 tablet_bootstrap.cc:654] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:49.891204 25208 tablet_bootstrap.cc:654] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:49.891518 25233 ts_tablet_manager.cc:1918] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:49.891621 25233 log.cc:1196] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/5b0bb07588a24da8a96121cb4f027046
I20260925 00:53:49.891919 25233 ts_tablet_manager.cc:1939] T 5b0bb07588a24da8a96121cb4f027046 P 1e2654fbd8a247499d319f44d8adab79: Deleting consensus metadata
I20260925 00:53:49.891942 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 66d124de700748eaa9833cd9467eb4b8 (DEFAULT_TABLE table=test-table [id=d000c6c3d63c4df6ab97ea246702199f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:49.892180 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d124de700748eaa9833cd9467eb4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:49.892647 24683 catalog_manager.cc:4841] TS 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705): tablet 5b0bb07588a24da8a96121cb4f027046 (table test-table [id=134e42fe4250431099d2d6cbb8f31c13]) successfully deleted
I20260925 00:53:49.892871 25231 ts_tablet_manager.cc:1918] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:49.893020 25231 log.cc:1196] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/5b0bb07588a24da8a96121cb4f027046
I20260925 00:53:49.893400 25231 ts_tablet_manager.cc:1939] T 5b0bb07588a24da8a96121cb4f027046 P 1aa9682f65a44e719f458211704e1a8e: Deleting consensus metadata
I20260925 00:53:49.893607 25208 tablet_bootstrap.cc:492] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: No bootstrap required, opened a new log
I20260925 00:53:49.893694 25208 ts_tablet_manager.cc:1397] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:49.894124 25208 raft_consensus.cc:348] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.894240 25208 raft_consensus.cc:374] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:49.894279 25208 raft_consensus.cc:729] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2654fbd8a247499d319f44d8adab79, State: Initialized, Role: FOLLOWER
I20260925 00:53:49.894397 25208 consensus_queue.cc:260] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.894656 25208 ts_tablet_manager.cc:1428] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:49.894842 24683 catalog_manager.cc:4841] TS 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429): tablet 5b0bb07588a24da8a96121cb4f027046 (table test-table [id=134e42fe4250431099d2d6cbb8f31c13]) successfully deleted
I20260925 00:53:49.894883 25210 tablet_bootstrap.cc:492] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: No bootstrap required, opened a new log
I20260925 00:53:49.894958 25210 ts_tablet_manager.cc:1397] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 00:53:49.894958 24682 catalog_manager.cc:4841] TS 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565): tablet 5b0bb07588a24da8a96121cb4f027046 (table test-table [id=134e42fe4250431099d2d6cbb8f31c13]) successfully deleted
I20260925 00:53:49.895066 25209 tablet_bootstrap.cc:492] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Bootstrap starting.
I20260925 00:53:49.895401 25210 raft_consensus.cc:348] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.895522 25210 raft_consensus.cc:374] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:49.895558 25210 raft_consensus.cc:729] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723536ea12274114b2035fa62ef99adf, State: Initialized, Role: FOLLOWER
I20260925 00:53:49.895695 25210 consensus_queue.cc:260] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.895937 25210 ts_tablet_manager.cc:1428] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:49.895989 25209 tablet_bootstrap.cc:654] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:49.897609 25209 tablet_bootstrap.cc:492] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: No bootstrap required, opened a new log
I20260925 00:53:49.897691 25209 ts_tablet_manager.cc:1397] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 00:53:49.898066 25209 raft_consensus.cc:348] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.898159 25209 raft_consensus.cc:374] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:49.898186 25209 raft_consensus.cc:729] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Initialized, Role: FOLLOWER
I20260925 00:53:49.898322 25209 consensus_queue.cc:260] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.898550 25209 ts_tablet_manager.cc:1428] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:49.997493 25214 raft_consensus.cc:482] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:49.997639 25214 raft_consensus.cc:504] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:49.998775 25214 leader_election.cc:288] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565), 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:50.003916 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.999042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41705 (local address 127.16.202.2:33747)
0925 00:53:49.999725 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.999737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.999822 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.000069 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.000071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.000084 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:50.000178 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.000184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.000973 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.000977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.002369 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.002381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.003411 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.003418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.003530 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.003555 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.003645 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.003733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":361,"thread_start_us":110,"threads_started":1}
I20260925 00:53:50.003890 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.999324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:33747 (local address 127.16.202.3:41705)
0925 00:53:49.999460 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.999464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.999821 (+ 357us) server_negotiation.cc:407] Connection header received
0925 00:53:49.999914 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.999917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.999970 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.000055 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:50.000296 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.000866 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.002622 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.003240 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.003403 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.003464 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.003693 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.003775 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:53:50.004952 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.999124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:58901 (local address 127.16.202.1:39565)
0925 00:53:49.999267 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:49.999272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:49.999493 (+ 221us) server_negotiation.cc:407] Connection header received
0925 00:53:49.999703 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:49.999707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:49.999760 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:49.999806 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:50.000236 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.001047 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.003550 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.004154 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.004304 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.004374 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.004748 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.004838 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:50.005096 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d124de700748eaa9833cd9467eb4b8" candidate_uuid: "1e2654fbd8a247499d319f44d8adab79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf" is_pre_election: true
I20260925 00:53:50.005270 25134 raft_consensus.cc:2454] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2654fbd8a247499d319f44d8adab79 in term 0.
I20260925 00:53:50.005646 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:49.999037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39565 (local address 127.16.202.2:58901)
0925 00:53:49.999398 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:49.999413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:49.999574 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:49.999956 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:49.999958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:49.999971 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:50.000080 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.000086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.001180 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.001183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.002369 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.002381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.004363 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.004367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.004471 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.004491 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.005181 (+ 690us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.005271 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":43,"thread_start_us":82,"threads_started":1}
I20260925 00:53:50.005667 24927 leader_election.cc:302] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2654fbd8a247499d319f44d8adab79, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.005987 25214 raft_consensus.cc:2781] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:50.006054 25214 raft_consensus.cc:482] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:50.006095 25214 raft_consensus.cc:3037] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.007154 25214 raft_consensus.cc:504] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.007539 25214 leader_election.cc:288] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [CANDIDATE]: Term 1 election: Requested vote from peers 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565), 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:50.007853 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d124de700748eaa9833cd9467eb4b8" candidate_uuid: "1e2654fbd8a247499d319f44d8adab79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa9682f65a44e719f458211704e1a8e" is_pre_election: true
I20260925 00:53:50.007962 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d124de700748eaa9833cd9467eb4b8" candidate_uuid: "1e2654fbd8a247499d319f44d8adab79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf"
I20260925 00:53:50.008076 25134 raft_consensus.cc:3037] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.008116 24854 raft_consensus.cc:2454] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2654fbd8a247499d319f44d8adab79 in term 0.
I20260925 00:53:50.008441 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d124de700748eaa9833cd9467eb4b8" candidate_uuid: "1e2654fbd8a247499d319f44d8adab79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa9682f65a44e719f458211704e1a8e"
I20260925 00:53:50.008539 24853 raft_consensus.cc:3037] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.009268 25134 raft_consensus.cc:2454] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2654fbd8a247499d319f44d8adab79 in term 1.
I20260925 00:53:50.009581 24927 leader_election.cc:302] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2654fbd8a247499d319f44d8adab79, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.009737 24853 raft_consensus.cc:2454] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2654fbd8a247499d319f44d8adab79 in term 1.
I20260925 00:53:50.009927 25214 raft_consensus.cc:2781] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:50.010140 25214 raft_consensus.cc:686] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2654fbd8a247499d319f44d8adab79, State: Running, Role: LEADER
I20260925 00:53:50.010413 25214 consensus_queue.cc:237] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.013118 24696 catalog_manager.cc:5495] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2). New cstate: current_term: 1 leader_uuid: "1e2654fbd8a247499d319f44d8adab79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } health_report { overall_health: UNKNOWN } } }
Sep 25 00:53:50 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.62@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:50.088222 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.074546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36985 (local address 127.0.0.1:55806)
0925 00:53:50.074844 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.074857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.074976 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.075199 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.075201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.075624 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:50.075719 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.075726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.076535 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.076538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.077130 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.077140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.077557 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.077560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.077663 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.083262 (+ 5599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:50.083279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:50.084943 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:50.086585 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:50.086598 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:50.086606 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:50.086843 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:50.087119 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:50.087128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:50.087131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:50.087206 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:50.087601 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:50.087609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:50.087724 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.087823 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.088004 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":5,"thread_start_us":86,"threads_started":1}
I20260925 00:53:50.088222 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.074614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.16.202.62:36985)
0925 00:53:50.074798 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.074802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.074932 (+ 130us) server_negotiation.cc:407] Connection header received
0925 00:53:50.075057 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.075060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.075098 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.075171 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:50.075835 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.076430 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.077235 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.077459 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.077544 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.085146 (+ 7602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:50.085149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:50.085151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:50.085176 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:50.086435 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:50.086941 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:50.086945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:50.086947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:50.087024 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:50.087288 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:50.087291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:50.087293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:50.087496 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:50.087585 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.087885 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.088073 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260925 00:53:50.093066 24696 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:55806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 00:53:50.093897 24696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:50.098701 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 41349647e4d54d2f93cf313725e1b92a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21b13f89088e45e4b447bff7e9b8f4f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:50.098755 25114 tablet_service.cc:1467] Processing CreateTablet for tablet 41349647e4d54d2f93cf313725e1b92a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21b13f89088e45e4b447bff7e9b8f4f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:50.098701 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 41349647e4d54d2f93cf313725e1b92a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21b13f89088e45e4b447bff7e9b8f4f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:53:50.099107 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41349647e4d54d2f93cf313725e1b92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.099112 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41349647e4d54d2f93cf313725e1b92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.099112 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41349647e4d54d2f93cf313725e1b92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.101670 25210 tablet_bootstrap.cc:492] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: Bootstrap starting.
I20260925 00:53:50.101966 25208 tablet_bootstrap.cc:492] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: Bootstrap starting.
I20260925 00:53:50.102002 25209 tablet_bootstrap.cc:492] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: Bootstrap starting.
I20260925 00:53:50.102648 25210 tablet_bootstrap.cc:654] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.102662 25209 tablet_bootstrap.cc:654] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.102823 25208 tablet_bootstrap.cc:654] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.103740 25209 tablet_bootstrap.cc:492] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: No bootstrap required, opened a new log
I20260925 00:53:50.103828 25209 ts_tablet_manager.cc:1397] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:50.104110 25208 tablet_bootstrap.cc:492] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: No bootstrap required, opened a new log
I20260925 00:53:50.104197 25208 ts_tablet_manager.cc:1397] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:50.104234 25209 raft_consensus.cc:348] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.104349 25209 raft_consensus.cc:374] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.104393 25209 raft_consensus.cc:729] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.104521 25209 consensus_queue.cc:260] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.104619 25208 raft_consensus.cc:348] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.104756 25208 raft_consensus.cc:374] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.104790 25208 raft_consensus.cc:729] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2654fbd8a247499d319f44d8adab79, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.104820 25209 ts_tablet_manager.cc:1428] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:50.104790 25216 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.104912 25208 consensus_queue.cc:260] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.104959 25216 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.105168 25208 ts_tablet_manager.cc:1428] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:50.105360 25210 tablet_bootstrap.cc:492] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: No bootstrap required, opened a new log
I20260925 00:53:50.105508 25210 ts_tablet_manager.cc:1397] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:53:50.105151 25214 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.105602 25214 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.105931 25210 raft_consensus.cc:348] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.106047 25210 raft_consensus.cc:374] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.106079 25210 raft_consensus.cc:729] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723536ea12274114b2035fa62ef99adf, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.106194 25210 consensus_queue.cc:260] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.106462 25210 ts_tablet_manager.cc:1428] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:50.106411 25215 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.106549 25215 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.257503 25216 raft_consensus.cc:482] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:50.257647 25216 raft_consensus.cc:504] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.258049 25216 leader_election.cc:288] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429), 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:50.258539 24993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41349647e4d54d2f93cf313725e1b92a" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79" is_pre_election: true
I20260925 00:53:50.258563 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41349647e4d54d2f93cf313725e1b92a" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf" is_pre_election: true
I20260925 00:53:50.258692 25134 raft_consensus.cc:2454] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 0.
I20260925 00:53:50.258694 24993 raft_consensus.cc:2454] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 0.
I20260925 00:53:50.259079 24787 leader_election.cc:302] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.259279 25216 raft_consensus.cc:2781] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:50.259339 25216 raft_consensus.cc:482] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:50.259384 25216 raft_consensus.cc:3037] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.260360 25216 raft_consensus.cc:504] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.260655 25216 leader_election.cc:288] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 election: Requested vote from peers 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429), 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:50.261054 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41349647e4d54d2f93cf313725e1b92a" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf"
I20260925 00:53:50.261059 24993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41349647e4d54d2f93cf313725e1b92a" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79"
I20260925 00:53:50.261169 24993 raft_consensus.cc:3037] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.261175 25134 raft_consensus.cc:3037] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.261996 25134 raft_consensus.cc:2454] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 1.
I20260925 00:53:50.262048 24993 raft_consensus.cc:2454] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 1.
I20260925 00:53:50.262274 24787 leader_election.cc:302] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.262441 25216 raft_consensus.cc:2781] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:50.262537 25216 raft_consensus.cc:686] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [term 1 LEADER]: Becoming Leader. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Running, Role: LEADER
I20260925 00:53:50.262678 25216 consensus_queue.cc:237] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.263351 25219 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: TxnStatusTablet state changed. Reason: New leader 1aa9682f65a44e719f458211704e1a8e. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.263455 25219 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:50.263728 25252 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 00:53:50.263881 25252 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 00:53:50.264428 24695 catalog_manager.cc:5495] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1aa9682f65a44e719f458211704e1a8e (127.16.202.1). New cstate: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:50.299365 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.295996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.16.202.1:39565)
0925 00:53:50.296126 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.296131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.296188 (+ 57us) server_negotiation.cc:407] Connection header received
0925 00:53:50.296378 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.296381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.296431 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.296494 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.296763 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.297290 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.298264 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.298489 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.298577 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.298889 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.298990 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.299053 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.299158 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.299216 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:53:50.299365 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.295872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39565 (local address 127.0.0.1:36048)
0925 00:53:50.296126 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.296142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.296261 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.296527 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.296530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.296538 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:50.296645 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.296650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.297397 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.297401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.298152 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.298163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.298593 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.298597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.298693 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.298741 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.299066 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.299085 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.299146 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.299242 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 00:53:50.302253 25134 raft_consensus.cc:1261] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Refusing update from remote peer 1aa9682f65a44e719f458211704e1a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.302237 24993 raft_consensus.cc:1261] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Refusing update from remote peer 1aa9682f65a44e719f458211704e1a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.302670 25216 consensus_queue.cc:1035] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.302850 25219 consensus_queue.cc:1035] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.304268 25215 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: TxnStatusTablet state changed. Reason: New leader 1aa9682f65a44e719f458211704e1a8e. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.304382 25215 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.304540 25214 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: TxnStatusTablet state changed. Reason: New leader 1aa9682f65a44e719f458211704e1a8e. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.304642 25214 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.306195 25219 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.306305 25219 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:50.306775 25222 mvcc.cc:205] Tried to move back new op lower bound from 7333059093713285120 to 7333059093557260288. Current Snapshot: MvccSnapshot[applied={T|T < 7333059093713285120 or (T in {7333059093713285120})}]
I20260925 00:53:50.307538 25214 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.307644 25214 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1e2654fbd8a247499d319f44d8adab79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.309016 25219 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.309124 25219 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 1aa9682f65a44e719f458211704e1a8e: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:53:50.309888 25215 tablet_replica.cc:439] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } } }
I20260925 00:53:50.310007 25215 tablet_replica.cc:442] T 41349647e4d54d2f93cf313725e1b92a P 723536ea12274114b2035fa62ef99adf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:53:50.316167 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.312182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.16.202.2:37429)
0925 00:53:50.312321 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.312326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.312340 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:53:50.312424 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.312427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.312490 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.312593 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.312993 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.313615 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.314756 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.314969 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.315039 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.315300 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.315762 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.315851 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.315954 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.316027 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:53:50.327832 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.323588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39565 (local address 127.16.202.2:34425)
0925 00:53:50.324005 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.324018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.324114 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.324428 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.324430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.324438 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:50.324537 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.324542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.325599 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.325602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.326377 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.326395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.327008 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.327012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.327102 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.327159 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.327491 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.327498 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.327567 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.327666 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":123,"threads_started":1}
I20260925 00:53:50.327822 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.323702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:34425 (local address 127.16.202.1:39565)
0925 00:53:50.323835 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.323839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.324096 (+ 257us) server_negotiation.cc:407] Connection header received
0925 00:53:50.324204 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.324207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.324265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.324347 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.324638 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.325483 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.326494 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.326865 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.326972 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.327290 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.327383 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.327482 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.327581 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.327647 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:53:50.335217 24853 raft_consensus.cc:1261] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Refusing update from remote peer 1e2654fbd8a247499d319f44d8adab79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.335717 25134 raft_consensus.cc:1261] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Refusing update from remote peer 1e2654fbd8a247499d319f44d8adab79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.335856 25214 consensus_queue.cc:1035] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.336115 25239 consensus_queue.cc:1035] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.362064 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.357991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34527 (local address 127.16.202.2:37429)
0925 00:53:50.358120 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.358125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.358421 (+ 296us) server_negotiation.cc:407] Connection header received
0925 00:53:50.358566 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.358570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.358627 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.358710 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.358971 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.359743 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.360781 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.361134 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.361231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.361533 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.361618 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.361683 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.361845 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.361918 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:53:50.362051 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.357885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37429 (local address 127.16.202.1:34527)
0925 00:53:50.358311 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.358341 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.358452 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.358737 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.358740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.358750 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:50.358866 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.358873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.359861 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.359865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.360618 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.360630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.361261 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.361266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.361344 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.361402 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.361722 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.361729 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.361799 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.361891 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":112,"threads_started":1}
I20260925 00:53:50.392916 24697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:50.393115 24697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:50.395649 24697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Sending DeleteTablet for 3 replicas of tablet 66d124de700748eaa9833cd9467eb4b8
I20260925 00:53:50.396307 24834 tablet_service.cc:1514] Processing DeleteTablet for tablet 66d124de700748eaa9833cd9467eb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:50 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36022
I20260925 00:53:50.396767 25267 tablet_replica.cc:330] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: stopping tablet replica
I20260925 00:53:50.397040 25267 raft_consensus.cc:2229] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:50.397218 25267 raft_consensus.cc:2258] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:50.396483 25114 tablet_service.cc:1514] Processing DeleteTablet for tablet 66d124de700748eaa9833cd9467eb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:50 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49812
I20260925 00:53:50.398079 25267 ts_tablet_manager.cc:1905] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:50.398165 24974 tablet_service.cc:1514] Processing DeleteTablet for tablet 66d124de700748eaa9833cd9467eb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:50 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:42518
I20260925 00:53:50.401360 25267 ts_tablet_manager.cc:1918] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:50.401508 25267 log.cc:1196] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/66d124de700748eaa9833cd9467eb4b8
I20260925 00:53:50.402024 25267 ts_tablet_manager.cc:1939] T 66d124de700748eaa9833cd9467eb4b8 P 1aa9682f65a44e719f458211704e1a8e: Deleting consensus metadata
I20260925 00:53:50.402714 25268 tablet_replica.cc:330] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: stopping tablet replica
I20260925 00:53:50.403118 25268 raft_consensus.cc:2229] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:50.403455 25268 raft_consensus.cc:2258] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:50.403991 24682 catalog_manager.cc:4841] TS 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565): tablet 66d124de700748eaa9833cd9467eb4b8 (table test-table [id=d000c6c3d63c4df6ab97ea246702199f]) successfully deleted
I20260925 00:53:50.404381 25268 ts_tablet_manager.cc:1905] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:50.404911 25269 tablet_replica.cc:330] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: stopping tablet replica
I20260925 00:53:50.405139 25269 raft_consensus.cc:2229] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:50.405637 25269 raft_consensus.cc:2258] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:50.406380 25269 ts_tablet_manager.cc:1905] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:50.407586 25268 ts_tablet_manager.cc:1918] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:50.407673 25268 log.cc:1196] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/66d124de700748eaa9833cd9467eb4b8
I20260925 00:53:50.407938 25268 ts_tablet_manager.cc:1939] T 66d124de700748eaa9833cd9467eb4b8 P 723536ea12274114b2035fa62ef99adf: Deleting consensus metadata
I20260925 00:53:50.408736 24683 catalog_manager.cc:4841] TS 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705): tablet 66d124de700748eaa9833cd9467eb4b8 (table test-table [id=d000c6c3d63c4df6ab97ea246702199f]) successfully deleted
I20260925 00:53:50.409454 25269 ts_tablet_manager.cc:1918] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:50.409562 25269 log.cc:1196] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/66d124de700748eaa9833cd9467eb4b8
I20260925 00:53:50.409832 25269 ts_tablet_manager.cc:1939] T 66d124de700748eaa9833cd9467eb4b8 P 1e2654fbd8a247499d319f44d8adab79: Deleting consensus metadata
I20260925 00:53:50.410586 24683 catalog_manager.cc:4841] TS 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429): tablet 66d124de700748eaa9833cd9467eb4b8 (table test-table [id=d000c6c3d63c4df6ab97ea246702199f]) successfully deleted
Sep 25 00:53:50 dist-test-slave-rpk1 krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:53:50 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:50 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:53:50.439601 25247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:50.433093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.16.202.62:36985)
0925 00:53:50.433223 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.433229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.433436 (+ 207us) server_negotiation.cc:407] Connection header received
0925 00:53:50.433570 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.433573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.433620 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.433686 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:50.434435 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.434969 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.435804 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.436036 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.436129 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.439469 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55822: BlockingRecv error: recv got EOF from 127.0.0.1:55822 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 00:53:50 dist-test-slave-rpk1 krb5kdc[24652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM
I20260925 00:53:50.450152 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.441545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.16.202.62:36985)
0925 00:53:50.441668 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.441672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.441781 (+ 109us) server_negotiation.cc:407] Connection header received
0925 00:53:50.441920 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.441923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.441976 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.442062 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:50.442629 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.443490 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.444326 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.444665 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.444790 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.447277 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:50.447281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:50.447284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:50.447342 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:50.448596 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:50.449084 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:50.449087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:50.449088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:50.449170 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:50.449415 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:50.449423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:50.449425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:50.449596 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:50.449694 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.449918 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.450019 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 00:53:50.452953 24697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:53:50.453444 24697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:50.458318 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 473ea61bb3964603b843e19dba9e10f6 (DEFAULT_TABLE table=test-table [id=9b2301e05af74cbf8ac8e80ce0d66fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:50.458482 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 473ea61bb3964603b843e19dba9e10f6 (DEFAULT_TABLE table=test-table [id=9b2301e05af74cbf8ac8e80ce0d66fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:50.458633 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473ea61bb3964603b843e19dba9e10f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.458760 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473ea61bb3964603b843e19dba9e10f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.460778 25208 tablet_bootstrap.cc:492] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Bootstrap starting.
I20260925 00:53:50.461001 25114 tablet_service.cc:1467] Processing CreateTablet for tablet 473ea61bb3964603b843e19dba9e10f6 (DEFAULT_TABLE table=test-table [id=9b2301e05af74cbf8ac8e80ce0d66fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:50.461254 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473ea61bb3964603b843e19dba9e10f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.461570 25208 tablet_bootstrap.cc:654] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.462563 25208 tablet_bootstrap.cc:492] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: No bootstrap required, opened a new log
I20260925 00:53:50.462647 25208 ts_tablet_manager.cc:1397] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:50.463032 25208 raft_consensus.cc:348] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.463127 25208 raft_consensus.cc:374] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.463156 25208 raft_consensus.cc:729] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2654fbd8a247499d319f44d8adab79, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.463287 25208 consensus_queue.cc:260] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.463537 25208 ts_tablet_manager.cc:1428] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:50.463549 25210 tablet_bootstrap.cc:492] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Bootstrap starting.
I20260925 00:53:50.464406 25210 tablet_bootstrap.cc:654] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.465147 25209 tablet_bootstrap.cc:492] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Bootstrap starting.
I20260925 00:53:50.465564 25210 tablet_bootstrap.cc:492] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: No bootstrap required, opened a new log
I20260925 00:53:50.465632 25210 ts_tablet_manager.cc:1397] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:50.465934 25209 tablet_bootstrap.cc:654] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.466056 25210 raft_consensus.cc:348] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.466166 25210 raft_consensus.cc:374] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.466197 25210 raft_consensus.cc:729] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723536ea12274114b2035fa62ef99adf, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.466305 25210 consensus_queue.cc:260] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.466564 25210 ts_tablet_manager.cc:1428] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:50.468176 25209 tablet_bootstrap.cc:492] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: No bootstrap required, opened a new log
I20260925 00:53:50.468278 25209 ts_tablet_manager.cc:1397] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 00:53:50.468614 25209 raft_consensus.cc:348] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.468750 25209 raft_consensus.cc:374] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.468787 25209 raft_consensus.cc:729] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.468912 25209 consensus_queue.cc:260] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.469156 25209 ts_tablet_manager.cc:1428] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:53:50.525312 25219 raft_consensus.cc:482] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:50.525470 25219 raft_consensus.cc:504] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.525965 25219 leader_election.cc:288] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429), 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:50.526453 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473ea61bb3964603b843e19dba9e10f6" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf" is_pre_election: true
I20260925 00:53:50.526464 24993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473ea61bb3964603b843e19dba9e10f6" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79" is_pre_election: true
I20260925 00:53:50.526590 25134 raft_consensus.cc:2454] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 0.
I20260925 00:53:50.526628 24993 raft_consensus.cc:2454] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 0.
I20260925 00:53:50.526898 24787 leader_election.cc:302] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 1e2654fbd8a247499d319f44d8adab79; no voters:
I20260925 00:53:50.527119 25219 raft_consensus.cc:2781] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:50.527189 25219 raft_consensus.cc:482] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:50.527248 25219 raft_consensus.cc:3037] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.528374 25219 raft_consensus.cc:504] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.528805 25219 leader_election.cc:288] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 election: Requested vote from peers 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429), 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705)
I20260925 00:53:50.529206 24993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473ea61bb3964603b843e19dba9e10f6" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79"
I20260925 00:53:50.529213 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473ea61bb3964603b843e19dba9e10f6" candidate_uuid: "1aa9682f65a44e719f458211704e1a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723536ea12274114b2035fa62ef99adf"
I20260925 00:53:50.529347 24993 raft_consensus.cc:3037] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.529407 25134 raft_consensus.cc:3037] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.530577 24993 raft_consensus.cc:2454] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 1.
I20260925 00:53:50.530584 25134 raft_consensus.cc:2454] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa9682f65a44e719f458211704e1a8e in term 1.
I20260925 00:53:50.530886 24787 leader_election.cc:302] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.531069 25219 raft_consensus.cc:2781] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:50.531175 25219 raft_consensus.cc:686] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 1 LEADER]: Becoming Leader. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Running, Role: LEADER
I20260925 00:53:50.531337 25219 consensus_queue.cc:237] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.533286 24695 catalog_manager.cc:5495] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1aa9682f65a44e719f458211704e1a8e (127.16.202.1). New cstate: current_term: 1 leader_uuid: "1aa9682f65a44e719f458211704e1a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } health_report { overall_health: UNKNOWN } } }
I20260925 00:53:50.554009 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.550382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.16.202.1:39565)
0925 00:53:50.550505 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.550508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.550523 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:50.550597 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.550599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.550648 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.550723 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.550994 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.551524 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.552905 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.553128 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.553214 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.553569 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.553651 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.553718 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.553804 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.553849 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:53:50.556173 24993 raft_consensus.cc:1261] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Refusing update from remote peer 1aa9682f65a44e719f458211704e1a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.556180 25134 raft_consensus.cc:1261] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Refusing update from remote peer 1aa9682f65a44e719f458211704e1a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.556617 25257 consensus_queue.cc:1035] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.556849 25219 consensus_queue.cc:1035] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.564916 24695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:50.565071 24695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:50.566932 24695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Sending DeleteTablet for 3 replicas of tablet 473ea61bb3964603b843e19dba9e10f6
I20260925 00:53:50.567445 24834 tablet_service.cc:1514] Processing DeleteTablet for tablet 473ea61bb3964603b843e19dba9e10f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:50 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36022
I20260925 00:53:50.567625 25267 tablet_replica.cc:330] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: stopping tablet replica
I20260925 00:53:50.567665 25114 tablet_service.cc:1514] Processing DeleteTablet for tablet 473ea61bb3964603b843e19dba9e10f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:50 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49812
I20260925 00:53:50.567776 25267 raft_consensus.cc:2229] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:50.567806 24974 tablet_service.cc:1514] Processing DeleteTablet for tablet 473ea61bb3964603b843e19dba9e10f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:50 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:42518
I20260925 00:53:50.567858 25268 tablet_replica.cc:330] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: stopping tablet replica
I20260925 00:53:50.567976 25269 tablet_replica.cc:330] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: stopping tablet replica
I20260925 00:53:50.567968 25267 raft_consensus.cc:2258] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:50.567979 25268 raft_consensus.cc:2229] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:50.568085 25269 raft_consensus.cc:2229] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:50.568105 25268 raft_consensus.cc:2258] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:50.568199 25269 raft_consensus.cc:2258] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:50.568439 24695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:53:50.568717 25267 ts_tablet_manager.cc:1905] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:50.568771 25268 ts_tablet_manager.cc:1905] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:53:50.568894 24695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:53:50.570996 25268 ts_tablet_manager.cc:1918] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:50.571065 25268 log.cc:1196] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/473ea61bb3964603b843e19dba9e10f6
I20260925 00:53:50.571058 25267 ts_tablet_manager.cc:1918] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:50.571131 25267 log.cc:1196] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/473ea61bb3964603b843e19dba9e10f6
I20260925 00:53:50.571300 25268 ts_tablet_manager.cc:1939] T 473ea61bb3964603b843e19dba9e10f6 P 723536ea12274114b2035fa62ef99adf: Deleting consensus metadata
I20260925 00:53:50.571367 25267 ts_tablet_manager.cc:1939] T 473ea61bb3964603b843e19dba9e10f6 P 1aa9682f65a44e719f458211704e1a8e: Deleting consensus metadata
I20260925 00:53:50.571982 24682 catalog_manager.cc:4841] TS 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565): tablet 473ea61bb3964603b843e19dba9e10f6 (table test-table [id=9b2301e05af74cbf8ac8e80ce0d66fd7]) successfully deleted
I20260925 00:53:50.572006 24683 catalog_manager.cc:4841] TS 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705): tablet 473ea61bb3964603b843e19dba9e10f6 (table test-table [id=9b2301e05af74cbf8ac8e80ce0d66fd7]) successfully deleted
I20260925 00:53:50.572572 25269 ts_tablet_manager.cc:1905] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:50.573628 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 925287cd000941c8a0fa0538447290a1 (DEFAULT_TABLE table=test-table [id=536db76f11e5454c9c0e03108213f62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:50.573657 25114 tablet_service.cc:1467] Processing CreateTablet for tablet 925287cd000941c8a0fa0538447290a1 (DEFAULT_TABLE table=test-table [id=536db76f11e5454c9c0e03108213f62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:50.573875 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925287cd000941c8a0fa0538447290a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.573894 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925287cd000941c8a0fa0538447290a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.574900 25269 ts_tablet_manager.cc:1918] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:53:50.574973 25269 log.cc:1196] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/473ea61bb3964603b843e19dba9e10f6
I20260925 00:53:50.575186 25269 ts_tablet_manager.cc:1939] T 473ea61bb3964603b843e19dba9e10f6 P 1e2654fbd8a247499d319f44d8adab79: Deleting consensus metadata
I20260925 00:53:50.575742 25210 tablet_bootstrap.cc:492] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Bootstrap starting.
I20260925 00:53:50.575850 25208 tablet_bootstrap.cc:492] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Bootstrap starting.
I20260925 00:53:50.575951 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 925287cd000941c8a0fa0538447290a1 (DEFAULT_TABLE table=test-table [id=536db76f11e5454c9c0e03108213f62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:53:50.576190 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925287cd000941c8a0fa0538447290a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:50.576473 25208 tablet_bootstrap.cc:654] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.576578 25210 tablet_bootstrap.cc:654] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.577419 25208 tablet_bootstrap.cc:492] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: No bootstrap required, opened a new log
I20260925 00:53:50.577488 25208 ts_tablet_manager.cc:1397] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:50.577843 25208 raft_consensus.cc:348] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.577936 25208 raft_consensus.cc:374] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.577957 25208 raft_consensus.cc:729] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2654fbd8a247499d319f44d8adab79, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.578064 25208 consensus_queue.cc:260] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.578312 25208 ts_tablet_manager.cc:1428] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:53:50.578322 25209 tablet_bootstrap.cc:492] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Bootstrap starting.
I20260925 00:53:50.578804 24683 catalog_manager.cc:4841] TS 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429): tablet 473ea61bb3964603b843e19dba9e10f6 (table test-table [id=9b2301e05af74cbf8ac8e80ce0d66fd7]) successfully deleted
I20260925 00:53:50.579113 25210 tablet_bootstrap.cc:492] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: No bootstrap required, opened a new log
I20260925 00:53:50.579155 25209 tablet_bootstrap.cc:654] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:50.579188 25210 ts_tablet_manager.cc:1397] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 00:53:50.579680 25210 raft_consensus.cc:348] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.579784 25210 raft_consensus.cc:374] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.579815 25210 raft_consensus.cc:729] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723536ea12274114b2035fa62ef99adf, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.579964 25210 consensus_queue.cc:260] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.580211 25209 tablet_bootstrap.cc:492] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: No bootstrap required, opened a new log
I20260925 00:53:50.580317 25209 ts_tablet_manager.cc:1397] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:53:50.580601 25210 ts_tablet_manager.cc:1428] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:53:50.581442 25209 raft_consensus.cc:348] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.581558 25209 raft_consensus.cc:374] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:50.581598 25209 raft_consensus.cc:729] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa9682f65a44e719f458211704e1a8e, State: Initialized, Role: FOLLOWER
I20260925 00:53:50.581715 25209 consensus_queue.cc:260] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.581943 25209 ts_tablet_manager.cc:1428] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 00:53:50.927774 25215 raft_consensus.cc:482] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:53:50.927945 25215 raft_consensus.cc:504] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.929180 25215 leader_election.cc:288] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565), 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429)
I20260925 00:53:50.933055 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.929493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:53465 (local address 127.16.202.1:39565)
0925 00:53:50.929625 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.929630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.929760 (+ 130us) server_negotiation.cc:407] Connection header received
0925 00:53:50.929983 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.929986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.930033 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.930090 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:50.930367 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.930924 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.932060 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.932451 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.932543 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.932598 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.932844 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.932920 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:53:50.933075 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.929838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:53869 (local address 127.16.202.2:37429)
0925 00:53:50.930052 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.930055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.930067 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:53:50.930116 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.930118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.930161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.930246 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:53:50.930499 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.930987 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.932081 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.932501 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.932616 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.932667 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.932872 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.932945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":60,"threads_started":1}
I20260925 00:53:50.933099 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.929425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39565 (local address 127.16.202.3:53465)
0925 00:53:50.929680 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.929692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.929854 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.930121 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.930123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.930139 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:50.930268 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.930275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.931047 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.931051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.931944 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.931953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.932582 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.932586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.932709 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.932750 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.932817 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.932921 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":26,"thread_start_us":82,"threads_started":1}
I20260925 00:53:50.933090 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.929466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37429 (local address 127.16.202.3:53869)
0925 00:53:50.929788 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.929800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.929878 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.930270 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.930273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.930286 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:53:50.930394 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.930400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.931095 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.931097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.931956 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.931965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.932618 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.932622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.932755 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.932773 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.932845 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.932929 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":69,"thread_start_us":63,"threads_started":1}
I20260925 00:53:50.933531 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925287cd000941c8a0fa0538447290a1" candidate_uuid: "723536ea12274114b2035fa62ef99adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa9682f65a44e719f458211704e1a8e" is_pre_election: true
I20260925 00:53:50.933706 24853 raft_consensus.cc:2454] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723536ea12274114b2035fa62ef99adf in term 0.
I20260925 00:53:50.933908 24993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925287cd000941c8a0fa0538447290a1" candidate_uuid: "723536ea12274114b2035fa62ef99adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79" is_pre_election: true
I20260925 00:53:50.934041 24993 raft_consensus.cc:2454] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723536ea12274114b2035fa62ef99adf in term 0.
I20260925 00:53:50.934103 25066 leader_election.cc:302] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.934522 25215 raft_consensus.cc:2781] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:53:50.934573 25215 raft_consensus.cc:482] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:53:50.934599 25215 raft_consensus.cc:3037] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.935549 25215 raft_consensus.cc:504] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.935887 25215 leader_election.cc:288] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [CANDIDATE]: Term 1 election: Requested vote from peers 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565), 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429)
I20260925 00:53:50.936244 24993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925287cd000941c8a0fa0538447290a1" candidate_uuid: "723536ea12274114b2035fa62ef99adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2654fbd8a247499d319f44d8adab79"
I20260925 00:53:50.936250 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925287cd000941c8a0fa0538447290a1" candidate_uuid: "723536ea12274114b2035fa62ef99adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa9682f65a44e719f458211704e1a8e"
I20260925 00:53:50.936376 24993 raft_consensus.cc:3037] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.936376 24853 raft_consensus.cc:3037] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:50.937536 24853 raft_consensus.cc:2454] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723536ea12274114b2035fa62ef99adf in term 1.
I20260925 00:53:50.937536 24993 raft_consensus.cc:2454] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723536ea12274114b2035fa62ef99adf in term 1.
I20260925 00:53:50.937847 25067 leader_election.cc:302] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1aa9682f65a44e719f458211704e1a8e, 1e2654fbd8a247499d319f44d8adab79, 723536ea12274114b2035fa62ef99adf; no voters:
I20260925 00:53:50.938019 25215 raft_consensus.cc:2781] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:50.938199 25215 raft_consensus.cc:686] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 1 LEADER]: Becoming Leader. State: Replica: 723536ea12274114b2035fa62ef99adf, State: Running, Role: LEADER
I20260925 00:53:50.938424 25215 consensus_queue.cc:237] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } }
I20260925 00:53:50.940778 24695 catalog_manager.cc:5495] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf reported cstate change: term changed from 0 to 1, leader changed from <none> to 723536ea12274114b2035fa62ef99adf (127.16.202.3). New cstate: current_term: 1 leader_uuid: "723536ea12274114b2035fa62ef99adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723536ea12274114b2035fa62ef99adf" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41705 } health_report { overall_health: HEALTHY } } }
I20260925 00:53:50.956597 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.953033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.16.202.3:41705)
0925 00:53:50.953313 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.953321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.953331 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:53:50.953395 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.953397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.953445 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.953553 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.953784 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.954335 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.955143 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.955375 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.955461 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.955734 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.956198 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.956303 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.956378 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.956432 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260925 00:53:50.966895 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.963846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:51321 (local address 127.16.202.1:39565)
0925 00:53:50.963958 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.963962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.963979 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:53:50.964124 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.964127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.964169 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.964227 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.964475 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.965009 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.965904 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.966112 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.966197 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.966478 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.966548 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.966603 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.966725 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.966775 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 00:53:50.966917 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.963636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39565 (local address 127.16.202.3:51321)
0925 00:53:50.963920 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.963935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.964030 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.964250 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.964259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.964267 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:50.964381 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.964386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.965106 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.965109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.965797 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.965805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.966217 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.966220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.966318 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.966360 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.966621 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.966627 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.966688 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.966762 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260925 00:53:50.972823 24853 raft_consensus.cc:1261] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Refusing update from remote peer 723536ea12274114b2035fa62ef99adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.972885 24993 raft_consensus.cc:1261] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Refusing update from remote peer 723536ea12274114b2035fa62ef99adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:53:50.973294 25215 consensus_queue.cc:1035] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2654fbd8a247499d319f44d8adab79" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.973510 25295 consensus_queue.cc:1035] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aa9682f65a44e719f458211704e1a8e" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:53:50.996141 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.992807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:35797 (local address 127.16.202.3:41705)
0925 00:53:50.992946 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:50.992951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:50.993138 (+ 187us) server_negotiation.cc:407] Connection header received
0925 00:53:50.993284 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:50.993286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:50.993414 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:50.993497 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:53:50.993742 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.994257 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.995050 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.995372 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.995470 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.995696 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.995774 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.995836 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:50.995976 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 00:53:50.996027 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:53:50.996182 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:50.992749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41705 (local address 127.16.202.1:35797)
0925 00:53:50.993070 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:50.993082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:50.993181 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:50.993526 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:50.993529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:50.993538 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:53:50.993652 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.993658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.994345 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.994347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.994945 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:50.994955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:50.995485 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:50.995490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:50.995559 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:50.995588 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:53:50.995871 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:53:50.995878 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:53:50.995943 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:53:50.996034 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260925 00:53:51.020933 24695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:51.021103 24695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:53:51.023394 24695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb7cc9af475d44ccbf1c3428642f2dcc: Sending DeleteTablet for 3 replicas of tablet 925287cd000941c8a0fa0538447290a1
I20260925 00:53:51.024091 24974 tablet_service.cc:1514] Processing DeleteTablet for tablet 925287cd000941c8a0fa0538447290a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:51 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:42518
I20260925 00:53:51.024116 24834 tablet_service.cc:1514] Processing DeleteTablet for tablet 925287cd000941c8a0fa0538447290a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:51 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36022
I20260925 00:53:51.024304 25269 tablet_replica.cc:330] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: stopping tablet replica
I20260925 00:53:51.024360 25267 tablet_replica.cc:330] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: stopping tablet replica
I20260925 00:53:51.024430 25269 raft_consensus.cc:2229] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:51.024492 25267 raft_consensus.cc:2229] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:53:51.024567 25269 raft_consensus.cc:2258] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:51.024618 25267 raft_consensus.cc:2258] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:51.024770 25114 tablet_service.cc:1514] Processing DeleteTablet for tablet 925287cd000941c8a0fa0538447290a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:53:51 UTC) from {username='oryx', principal='oryx/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49812
I20260925 00:53:51.024938 25268 tablet_replica.cc:330] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: stopping tablet replica
I20260925 00:53:51.025063 25268 raft_consensus.cc:2229] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:53:51.025363 25268 raft_consensus.cc:2258] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:53:51.025543 25267 ts_tablet_manager.cc:1905] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:51.026015 25269 ts_tablet_manager.cc:1905] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:51.027999 25269 ts_tablet_manager.cc:1918] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:51.028055 25269 log.cc:1196] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/925287cd000941c8a0fa0538447290a1
I20260925 00:53:51.028218 25269 ts_tablet_manager.cc:1939] T 925287cd000941c8a0fa0538447290a1 P 1e2654fbd8a247499d319f44d8adab79: Deleting consensus metadata
I20260925 00:53:51.028213 25267 ts_tablet_manager.cc:1918] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:51.028280 25267 log.cc:1196] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/925287cd000941c8a0fa0538447290a1
I20260925 00:53:51.028506 25267 ts_tablet_manager.cc:1939] T 925287cd000941c8a0fa0538447290a1 P 1aa9682f65a44e719f458211704e1a8e: Deleting consensus metadata
I20260925 00:53:51.028825 25268 ts_tablet_manager.cc:1905] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:53:51.029340 24683 catalog_manager.cc:4841] TS 1e2654fbd8a247499d319f44d8adab79 (127.16.202.2:37429): tablet 925287cd000941c8a0fa0538447290a1 (table test-table [id=536db76f11e5454c9c0e03108213f62d]) successfully deleted
I20260925 00:53:51.029685 24682 catalog_manager.cc:4841] TS 1aa9682f65a44e719f458211704e1a8e (127.16.202.1:39565): tablet 925287cd000941c8a0fa0538447290a1 (table test-table [id=536db76f11e5454c9c0e03108213f62d]) successfully deleted
I20260925 00:53:51.030112 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 24769
I20260925 00:53:51.031278 25268 ts_tablet_manager.cc:1918] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:53:51.031342 25268 log.cc:1196] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/925287cd000941c8a0fa0538447290a1
I20260925 00:53:51.031556 25268 ts_tablet_manager.cc:1939] T 925287cd000941c8a0fa0538447290a1 P 723536ea12274114b2035fa62ef99adf: Deleting consensus metadata
I20260925 00:53:51.032299 24683 catalog_manager.cc:4841] TS 723536ea12274114b2035fa62ef99adf (127.16.202.3:41705): tablet 925287cd000941c8a0fa0538447290a1 (table test-table [id=536db76f11e5454c9c0e03108213f62d]) successfully deleted
I20260925 00:53:51.037921 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 24909
I20260925 00:53:51.044023 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25049
I20260925 00:53:51.049919 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 24665
2026-09-25T00:53:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3592 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:51 dist-test-slave-rpk1 krb5kdc[25310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:51 dist-test-slave-rpk1 krb5kdc[25310](info): set up 2 sockets
Sep 25 00:53:51 dist-test-slave-rpk1 krb5kdc[25310](info): commencing operation
krb5kdc: starting...
W20260925 00:53:52.021330 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:52 dist-test-slave-rpk1 krb5kdc[25310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:52.145021 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34249
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:52.256379 25323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:52.256704 25323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:52.256762 25323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:52.260921 25323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:52.260983 25323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:52.261003 25323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:52.261021 25323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:52.261050 25323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:52.266641 25323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:34249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:52.267755 25323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:52.268868 25323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:52.277344 25329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:52.277441 25323 server_base.cc:1034] running on GCE node
W20260925 00:53:52.277344 25331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:52.277415 25328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:52.278277 25323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:52.279424 25323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:52.280609 25323 hybrid_clock.cc:648] HybridClock initialized: now 1790297632280585 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T00:53:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1243 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:53:52 dist-test-slave-rpk1 krb5kdc[25336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:53:52 dist-test-slave-rpk1 krb5kdc[25336](info): set up 2 sockets
Sep 25 00:53:52 dist-test-slave-rpk1 krb5kdc[25336](info): commencing operation
krb5kdc: starting...
W20260925 00:53:53.249343 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:53:53 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:53:53.375847 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40467
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:53.490018 25349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:53.490338 25349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:53.490401 25349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:53.494601 25349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:53.494679 25349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:53.494709 25349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:53.494730 25349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:53.494760 25349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:53.500408 25349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:40467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:53.501605 25349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:53.502658 25349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:53.511417 25354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:53.511421 25355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:53.511438 25357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:53.511996 25349 server_base.cc:1034] running on GCE node
I20260925 00:53:53.512725 25349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:53.513906 25349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:53.515072 25349 hybrid_clock.cc:648] HybridClock initialized: now 1790297633515054 us; error 33 us; skew 500 ppm
Sep 25 00:53:53 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:53.517879 25349 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:53.518831 25349 webserver.cc:460] Webserver started at http://127.16.202.62:37487/ using document root <none> and password file <none>
I20260925 00:53:53.519392 25349 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:53.519496 25349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:53.519711 25349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:53.521999 25349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "6838224158614a4380f4d4d646ea6c6c"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "660c14a9878cf1f5a9a9d905123d33cb"
server_key_iv: "15ec26119c532bde5b25659f89f16ee5"
server_key_version: "encryptionkey@0"
I20260925 00:53:53.522583 25349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "6838224158614a4380f4d4d646ea6c6c"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "660c14a9878cf1f5a9a9d905123d33cb"
server_key_iv: "15ec26119c532bde5b25659f89f16ee5"
server_key_version: "encryptionkey@0"
I20260925 00:53:53.526608 25349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:53:53.529358 25364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:53.530722 25349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:53.530869 25349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "6838224158614a4380f4d4d646ea6c6c"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "660c14a9878cf1f5a9a9d905123d33cb"
server_key_iv: "15ec26119c532bde5b25659f89f16ee5"
server_key_version: "encryptionkey@0"
I20260925 00:53:53.530994 25349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:53.543286 25349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:53.546674 25349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:53.546864 25349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:53.553905 25349 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:40467
I20260925 00:53:53.553932 25416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:40467 every 8 connection(s)
I20260925 00:53:53.554862 25349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:53.557685 25417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:53:53.562681 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25349
I20260925 00:53:53.562834 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:53:53.563140 17192 external_mini_cluster.cc:1428] Setting key 4c263e83ada6dbdf8383f32f381719e1
I20260925 00:53:53.563643 25417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Bootstrap starting.
I20260925 00:53:53.566191 25417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Neither blocks nor log segments found. Creating new log.
I20260925 00:53:53.567121 25417 log.cc:824] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:53.569372 25417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: No bootstrap required, opened a new log
Sep 25 00:53:53 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:53.573643 25417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:53.573890 25417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:53:53.573949 25417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6838224158614a4380f4d4d646ea6c6c, State: Initialized, Role: FOLLOWER
I20260925 00:53:53.574417 25417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:53.574553 25417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:53.574620 25417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:53.574718 25417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:53:53.575589 25417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:53.575930 25417 leader_election.cc:302] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6838224158614a4380f4d4d646ea6c6c; no voters:
I20260925 00:53:53.576227 25417 leader_election.cc:288] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:53:53.576318 25422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:53:53.576496 25422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 LEADER]: Becoming Leader. State: Replica: 6838224158614a4380f4d4d646ea6c6c, State: Running, Role: LEADER
I20260925 00:53:53.576836 25422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:53.577606 25417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:53.578097 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:53.564573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.16.202.62:40467)
0925 00:53:53.565032 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:53.565049 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:53.565085 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:53:53.565823 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:53.565843 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:53.566199 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:53.566533 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:53.567382 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:53.568369 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:53.569555 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:53.570064 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:53.570195 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:53.572995 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:53.573001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:53.573016 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:53.573044 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:53.574726 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:53.575156 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:53.575163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:53.575170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:53.575287 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:53.575547 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:53.575551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:53.575553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:53.576001 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:53.576189 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:53.576950 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 00:53:53.577231 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":131,"threads_started":1}
I20260925 00:53:53.578431 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6838224158614a4380f4d4d646ea6c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } } }
I20260925 00:53:53.578563 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:53.578879 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6838224158614a4380f4d4d646ea6c6c. Latest consensus state: current_term: 1 leader_uuid: "6838224158614a4380f4d4d646ea6c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } } }
I20260925 00:53:53.578969 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:53.578935 25431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:53.582033 25431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:53.587895 25431 catalog_manager.cc:1347] Generated new cluster ID: 98012a9218bc47dba759f44daa17f95a
I20260925 00:53:53.587980 25431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:53.602010 25431 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:53:53.603327 25431 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:53.611409 25431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Generated new TSK 0
I20260925 00:53:53.612315 25431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:53:53.645826 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40467
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:53.760725 25443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:53.761112 25443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:53.761180 25443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:53.766088 25443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:53.766170 25443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:53.766265 25443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:53.771883 25443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:53.773038 25443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:53.774080 25443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:53.783774 25451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:53.783775 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:53.783799 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:53.784137 25443 server_base.cc:1034] running on GCE node
I20260925 00:53:53.784564 25443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:53.785267 25443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:53.786427 25443 hybrid_clock.cc:648] HybridClock initialized: now 1790297633786407 us; error 32 us; skew 500 ppm
Sep 25 00:53:53 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:53.789130 25443 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:53:53.789985 25443 webserver.cc:460] Webserver started at http://127.16.202.1:41993/ using document root <none> and password file <none>
I20260925 00:53:53.790498 25443 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:53.790551 25443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:53.790714 25443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:53.792738 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "ee00f7b06c9649698ab9882ef5ad81a1"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "84d29f08096f640b64fe5a1510b38922"
server_key_iv: "a4332dd59d0955f0a61c03062808deef"
server_key_version: "encryptionkey@0"
I20260925 00:53:53.793242 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "ee00f7b06c9649698ab9882ef5ad81a1"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "84d29f08096f640b64fe5a1510b38922"
server_key_iv: "a4332dd59d0955f0a61c03062808deef"
server_key_version: "encryptionkey@0"
I20260925 00:53:53.796943 25443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:53:53.799525 25458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:53.800997 25443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:53.801111 25443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "ee00f7b06c9649698ab9882ef5ad81a1"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "84d29f08096f640b64fe5a1510b38922"
server_key_iv: "a4332dd59d0955f0a61c03062808deef"
server_key_version: "encryptionkey@0"
I20260925 00:53:53.801210 25443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:53.816135 25443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:53.819847 25443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:53.820034 25443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:53.820714 25443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:53.821655 25443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:53.821704 25443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:53.821746 25443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:53.821760 25443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:53.831019 25443 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43651
I20260925 00:53:53.831037 25571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43651 every 8 connection(s)
I20260925 00:53:53.831995 25443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:53.832966 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25443
I20260925 00:53:53.833101 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:53:53.833425 17192 external_mini_cluster.cc:1428] Setting key aef8b52223454e214ed4703f3a99a308
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
Sep 25 00:53:53 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:53:53.847000 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:53.834231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:60623 (local address 127.16.202.62:40467)
0925 00:53:53.834344 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:53.834347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:53.835652 (+ 1305us) server_negotiation.cc:407] Connection header received
0925 00:53:53.836791 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:53.836795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:53.836849 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:53.836954 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:53.838162 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:53.838934 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:53.840165 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:53.840529 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:53.840646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:53.843346 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:53.843349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:53.843352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:53.843388 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:53.844759 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:53.845343 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:53.845347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:53.845348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:53.845440 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:53.845818 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:53.845821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:53.845823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:53.846053 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:53.846149 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:53.846702 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 00:53:53.846831 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 00:53:53.847860 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:53.834638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:60623)
0925 00:53:53.835500 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:53.835556 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:53.836529 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:53.837124 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:53.837131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:53.837489 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:53.838000 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:53.838015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:53.839086 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:53.839090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:53.840040 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:53.840050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:53.840756 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:53.840761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:53.840936 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:53.841517 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:53.841533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:53.843130 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:53.844886 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:53.844892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:53.844907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:53.845181 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:53.845527 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:53.845530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:53.845532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:53.845718 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:53.846156 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:53.846162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:53.846460 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 00:53:53.846628 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 00:53:53.846920 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":137,"threads_started":1}
I20260925 00:53:53.849155 25572 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40467
I20260925 00:53:53.849443 25572 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:53.850077 25572 heartbeater.cc:511] Master 127.16.202.62:40467 requested a full tablet report, sending...
I20260925 00:53:53.851966 25381 ts_manager.cc:194] Registered new tserver with Master: ee00f7b06c9649698ab9882ef5ad81a1 (127.16.202.1:43651)
I20260925 00:53:53.853595 25381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:60623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:53:53.863651 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40467
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:53.976938 25577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:53.977227 25577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:53.977288 25577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:53.981552 25577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:53.981619 25577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:53.981704 25577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:53.987176 25577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:53.988329 25577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:53.989429 25577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:53.998802 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:53.998816 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:53.998802 25583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:53.999058 25577 server_base.cc:1034] running on GCE node
I20260925 00:53:53.999536 25577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:54.000161 25577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:54.001323 25577 hybrid_clock.cc:648] HybridClock initialized: now 1790297634001299 us; error 35 us; skew 500 ppm
Sep 25 00:53:53 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:54.003903 25577 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:53:54.004817 25577 webserver.cc:460] Webserver started at http://127.16.202.2:39077/ using document root <none> and password file <none>
I20260925 00:53:54.005388 25577 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:54.005446 25577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:54.005609 25577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:54.007575 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "5cb901ab7df6481290d59f6492acfdd3"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "5428592a564968434b593146e7b84a93"
server_key_iv: "2c7f3c19d5c2681ad2fcdb49c35735ce"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.008071 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "5cb901ab7df6481290d59f6492acfdd3"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "5428592a564968434b593146e7b84a93"
server_key_iv: "2c7f3c19d5c2681ad2fcdb49c35735ce"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.011955 25577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:53:54.014868 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.016502 25577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:54.016654 25577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "5cb901ab7df6481290d59f6492acfdd3"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "5428592a564968434b593146e7b84a93"
server_key_iv: "2c7f3c19d5c2681ad2fcdb49c35735ce"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.016819 25577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:54.033434 25577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:54.036427 25577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:54.036598 25577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:54.037215 25577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:54.038033 25577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:54.038080 25577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.038121 25577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:54.038149 25577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.046569 25577 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:46675
I20260925 00:53:54.046612 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:46675 every 8 connection(s)
I20260925 00:53:54.047523 25577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:54.050058 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25577
I20260925 00:53:54.050201 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:53:54.050593 17192 external_mini_cluster.cc:1428] Setting key 7e0273007c63426961731b6ccd9260b9
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:53:54.062646 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.049432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:55189 (local address 127.16.202.62:40467)
0925 00:53:54.049584 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:54.049589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:54.050885 (+ 1296us) server_negotiation.cc:407] Connection header received
0925 00:53:54.051941 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:54.051944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:54.051993 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:54.052215 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:54.053704 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.054446 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.055777 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.056097 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.056209 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.059170 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:54.059173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:54.059175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:54.059203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:54.060624 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:54.061209 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:54.061212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:54.061213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:54.061293 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:54.061620 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:54.061622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:54.061623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:54.061802 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:54.061908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:54.062384 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 00:53:54.062510 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:53:54.063630 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.049867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:55189)
0925 00:53:54.050693 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:54.050765 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:54.051616 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:54.052437 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:54.052445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:54.052863 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:54.053407 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.053526 (+ 119us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.054599 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.054602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.055627 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.055638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.056277 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.056281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.056524 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.057239 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:54.057257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:54.058930 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:54.060771 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:54.060776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:54.060790 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:54.061112 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:54.061365 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:54.061367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:54.061368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:54.061551 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:54.061884 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:54.061890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:54.062171 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 00:53:54.062332 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 00:53:54.062741 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":106,"threads_started":1}
I20260925 00:53:54.064936 25706 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40467
I20260925 00:53:54.065224 25706 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:54.065809 25706 heartbeater.cc:511] Master 127.16.202.62:40467 requested a full tablet report, sending...
I20260925 00:53:54.067000 25381 ts_manager.cc:194] Registered new tserver with Master: 5cb901ab7df6481290d59f6492acfdd3 (127.16.202.2:46675)
I20260925 00:53:54.067601 25381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:55189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:53:54.079481 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40467
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:54.193153 25711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:54.193450 25711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:54.193495 25711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:54.197651 25711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:54.197721 25711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:54.197784 25711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:54.203119 25711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:54.204160 25711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:54.205293 25711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:54.215509 25717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:54.215512 25719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:54.215512 25716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:54.215760 25711 server_base.cc:1034] running on GCE node
I20260925 00:53:54.216323 25711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:54.217072 25711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:54.218251 25711 hybrid_clock.cc:648] HybridClock initialized: now 1790297634218204 us; error 60 us; skew 500 ppm
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:54.221246 25711 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:53:54.222342 25711 webserver.cc:460] Webserver started at http://127.16.202.3:44551/ using document root <none> and password file <none>
I20260925 00:53:54.222970 25711 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:54.223044 25711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:54.223261 25711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:53:54.225584 25711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "7a7693c64d9d43288a2acfebebacf405"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "cef1daf6ac6370ccf904aa46971c279e"
server_key_iv: "fc3fccf7279d8412231b3e18a83ce684"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.226154 25711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "7a7693c64d9d43288a2acfebebacf405"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "cef1daf6ac6370ccf904aa46971c279e"
server_key_iv: "fc3fccf7279d8412231b3e18a83ce684"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.230116 25711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:53:54.232923 25726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.234274 25711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:54.234431 25711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "7a7693c64d9d43288a2acfebebacf405"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "cef1daf6ac6370ccf904aa46971c279e"
server_key_iv: "fc3fccf7279d8412231b3e18a83ce684"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.234544 25711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:54.266595 25711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:54.269907 25711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:54.270094 25711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:54.270706 25711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:54.271552 25711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:54.271601 25711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.271656 25711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:54.271680 25711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.280424 25711 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:35173
I20260925 00:53:54.280447 25839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:35173 every 8 connection(s)
I20260925 00:53:54.281467 25711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:54.286477 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25711
I20260925 00:53:54.286612 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:53:54.286882 17192 external_mini_cluster.cc:1428] Setting key e4dbf0dc86495ae6d32e806cbd360db4
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:53:54.295039 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.283439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:51057 (local address 127.16.202.62:40467)
0925 00:53:54.283567 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:54.283570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:54.284692 (+ 1122us) server_negotiation.cc:407] Connection header received
0925 00:53:54.285622 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:54.285625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:54.285685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:54.285764 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:54.287043 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.287680 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.289203 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.289426 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.289506 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.291829 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:54.291832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:54.291834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:54.291859 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:54.293275 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:54.293728 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:54.293730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:54.293732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:54.293809 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:54.294079 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:54.294081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:54.294082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:54.294246 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:54.294316 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:54.294813 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 00:53:54.294913 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:53:54.295969 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.283767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:51057)
0925 00:53:54.284484 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:54.284534 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:54.285399 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:54.285970 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:54.285978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:54.286351 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:54.286890 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.286905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.287811 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.287815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.289091 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.289102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.289549 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.289552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.289695 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.290275 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:54.290291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:54.291687 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:53:54.293380 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:54.293385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:54.293399 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:54.293651 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:54.293877 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:53:54.293879 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:53:54.293880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:53:54.294013 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:53:54.294360 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:53:54.294366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:53:54.294611 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 00:53:54.294787 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 00:53:54.295073 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":131,"threads_started":1}
I20260925 00:53:54.297284 25840 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40467
I20260925 00:53:54.297569 25840 heartbeater.cc:464] Registering TS with master...
I20260925 00:53:54.298046 25840 heartbeater.cc:511] Master 127.16.202.62:40467 requested a full tablet report, sending...
I20260925 00:53:54.299232 25381 ts_manager.cc:194] Registered new tserver with Master: 7a7693c64d9d43288a2acfebebacf405 (127.16.202.3:35173)
I20260925 00:53:54.299929 25381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:51057
I20260925 00:53:54.301064 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab.
Entry for principal oryx/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab.
WARNING: no policy specified for oryx/127.16.202.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab.
Entry for principal oryx/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab.
WARNING: no policy specified for oryx/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.202.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab.
Entry for principal oryx/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab.
I20260925 00:53:54.376178 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25443
I20260925 00:53:54.381171 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25577
I20260925 00:53:54.386277 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25711
I20260925 00:53:54.390795 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25349
I20260925 00:53:54.396082 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40467
--webserver_interface=127.16.202.62
--webserver_port=37487
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:53:54.506306 25849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:54.506604 25849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:54.506647 25849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:54.510825 25849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:53:54.510903 25849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:54.510922 25849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:54.510939 25849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:53:54.510953 25849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:53:54.516638 25849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:40467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=37487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:54.517834 25849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:54.518888 25849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:54.527760 25854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:54.527874 25849 server_base.cc:1034] running on GCE node
W20260925 00:53:54.527756 25855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:54.527776 25857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:54.528759 25849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:54.530006 25849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:54.531232 25849 hybrid_clock.cc:648] HybridClock initialized: now 1790297634531204 us; error 50 us; skew 500 ppm
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297634, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:54.534688 25849 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:53:54.535668 25849 webserver.cc:460] Webserver started at http://127.16.202.62:37487/ using document root <none> and password file <none>
I20260925 00:53:54.536239 25849 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:54.536324 25849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:54.540560 25849 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:54.543030 25864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.544374 25849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:53:54.544557 25849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "6838224158614a4380f4d4d646ea6c6c"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "660c14a9878cf1f5a9a9d905123d33cb"
server_key_iv: "15ec26119c532bde5b25659f89f16ee5"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.545250 25849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:54.558967 25849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:54.562423 25849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:54.562630 25849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:54.570897 25849 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:40467
I20260925 00:53:54.570941 25916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:40467 every 8 connection(s)
I20260925 00:53:54.572081 25849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:53:54.572476 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25849
I20260925 00:53:54.572973 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:43651
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:40467
--webserver_port=41993
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:53:54.577523 25917 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:53:54.580226 25917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Bootstrap starting.
I20260925 00:53:54.603176 25917 log.cc:824] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:53:54.611505 25917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:53:54.611919 25917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Bootstrap complete.
I20260925 00:53:54.617444 25917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:54.617982 25917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6838224158614a4380f4d4d646ea6c6c, State: Initialized, Role: FOLLOWER
I20260925 00:53:54.618556 25917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:54.618701 25917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:53:54.618767 25917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:53:54.618872 25917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:53:54.621387 25917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:54.621767 25917 leader_election.cc:302] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6838224158614a4380f4d4d646ea6c6c; no voters:
I20260925 00:53:54.622110 25917 leader_election.cc:288] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:53:54.622277 25921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:53:54.622540 25921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [term 2 LEADER]: Becoming Leader. State: Replica: 6838224158614a4380f4d4d646ea6c6c, State: Running, Role: LEADER
I20260925 00:53:54.622856 25921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } }
I20260925 00:53:54.623420 25917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:53:54.624162 25922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6838224158614a4380f4d4d646ea6c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } } }
I20260925 00:53:54.624300 25922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:54.624629 25923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6838224158614a4380f4d4d646ea6c6c. Latest consensus state: current_term: 2 leader_uuid: "6838224158614a4380f4d4d646ea6c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6838224158614a4380f4d4d646ea6c6c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40467 } } }
I20260925 00:53:54.624740 25923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c [sys.catalog]: This master's current role is: LEADER
I20260925 00:53:54.625458 25930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:53:54.628439 25930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:53:54.629033 25930 catalog_manager.cc:1259] Loaded cluster ID: 98012a9218bc47dba759f44daa17f95a
I20260925 00:53:54.629093 25930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:53:54.630846 25930 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:53:54.631408 25930 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6838224158614a4380f4d4d646ea6c6c: Loaded TSK: 0
I20260925 00:53:54.632369 25930 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:53:54.717849 25919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:54.718148 25919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:54.718219 25919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:54.722515 25919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:54.722610 25919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:54.722713 25919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:53:54.729005 25919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.1.keytab
--principal=oryx/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=41993
--tserver_master_addrs=127.16.202.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:54.730113 25919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:54.731192 25919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:54.741425 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:54.741508 25943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:54.741818 25946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:54.742084 25919 server_base.cc:1034] running on GCE node
I20260925 00:53:54.742506 25919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:54.743224 25919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:54.744501 25919 hybrid_clock.cc:648] HybridClock initialized: now 1790297634744480 us; error 31 us; skew 500 ppm
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297634, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:54.747673 25919 init.cc:377] Logged in from keytab as oryx/127.16.202.1@KRBTEST.COM (short username oryx)
I20260925 00:53:54.748939 25919 webserver.cc:460] Webserver started at http://127.16.202.1:41993/ using document root <none> and password file <none>
I20260925 00:53:54.749691 25919 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:54.749810 25919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:54.755278 25919 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:53:54.758025 25953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.759557 25919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:53:54.759686 25919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "ee00f7b06c9649698ab9882ef5ad81a1"
format_stamp: "Formatted at 2026-09-25 00:53:53 on dist-test-slave-rpk1"
server_key: "84d29f08096f640b64fe5a1510b38922"
server_key_iv: "a4332dd59d0955f0a61c03062808deef"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.760387 25919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:54.784031 25919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:54.788040 25919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:54.788254 25919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:54.788985 25919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:54.789801 25919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:54.789845 25919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.789901 25919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:54.789920 25919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.798687 25919 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43651
I20260925 00:53:54.798767 26066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43651 every 8 connection(s)
I20260925 00:53:54.799741 25919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:53:54.802651 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 25919
I20260925 00:53:54.803143 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:46675
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:40467
--webserver_port=39077
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:54.813606 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.802048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:45065)
0925 00:53:54.802699 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:54.802740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:54.803611 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:54.804751 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:54.804759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:54.805192 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:54.805799 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.805817 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.807115 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.807119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.808496 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.808508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.809274 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.809279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.809476 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.810093 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:54.810115 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:54.812588 (+ 2473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":112,"threads_started":1}
W20260925 00:53:54.813596 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:54.801935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:45065 (local address 127.16.202.62:40467)
0925 00:53:54.802468 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:54.802489 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:54.802874 (+ 385us) server_negotiation.cc:407] Connection header received
0925 00:53:54.804016 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:54.804027 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:54.804326 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:54.804632 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:54.805989 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.806977 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.808627 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.809119 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.809237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.812861 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:45065: BlockingRecv error: recv got EOF from 127.16.202.1:45065 (error 108)
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":127,"threads_started":1}
W20260925 00:53:54.814064 26067 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:40467 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.202.62:40467: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:54.819515 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.814486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:51959)
0925 00:53:54.814800 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:54.814829 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:54.814917 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:54.815138 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:54.815140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:54.815288 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:54.815376 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.815380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.816080 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.816083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.816649 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.816657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.817107 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.817110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.817200 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.817635 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:54.817646 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:54.819375 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260925 00:53:54.819624 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:54.814693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:51959 (local address 127.16.202.62:40467)
0925 00:53:54.814793 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:54.814798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:54.814891 (+ 93us) server_negotiation.cc:407] Connection header received
0925 00:53:54.815004 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:54.815006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:54.815049 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:54.815113 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:54.815488 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.815987 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.816770 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.817009 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.817094 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.819524 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:51959: BlockingRecv error: recv got EOF from 127.16.202.1:51959 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:54.825169 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.820050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:44119)
0925 00:53:54.820196 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:54.820210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:54.820280 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:54.820530 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:54.820533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:54.820698 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:54.820783 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.820788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.821465 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.821468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.821984 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.821991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.822374 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.822377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.822449 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.823067 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:54.823079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:54.825027 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 00:53:54.825291 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:54.820155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:44119 (local address 127.16.202.62:40467)
0925 00:53:54.820269 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:54.820272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:54.820285 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:54.820354 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:54.820357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:54.820436 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:54.820501 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:54.820879 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.821364 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.822081 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.822281 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.822362 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:54.825182 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:44119: BlockingRecv error: recv got EOF from 127.16.202.1:44119 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 00:53:54.825443 26067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:53:54.916379 26071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:54.916657 26071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:54.916742 26071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:54.920848 26071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:54.920902 26071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:54.920969 26071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:53:54.926308 26071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.2.keytab
--principal=oryx/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:46675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=39077
--tserver_master_addrs=127.16.202.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:54.927296 26071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:54.928285 26071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:54.938167 26076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:54.938172 26077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:54.938320 26071 server_base.cc:1034] running on GCE node
W20260925 00:53:54.938172 26079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:54.938932 26071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:54.939606 26071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:54.940770 26071 hybrid_clock.cc:648] HybridClock initialized: now 1790297634940741 us; error 41 us; skew 500 ppm
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297634, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:54.943528 26071 init.cc:377] Logged in from keytab as oryx/127.16.202.2@KRBTEST.COM (short username oryx)
I20260925 00:53:54.944432 26071 webserver.cc:460] Webserver started at http://127.16.202.2:39077/ using document root <none> and password file <none>
I20260925 00:53:54.945041 26071 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:54.945122 26071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:54.948932 26071 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 00:53:54.951009 26086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.952382 26071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:53:54.952514 26071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "5cb901ab7df6481290d59f6492acfdd3"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "5428592a564968434b593146e7b84a93"
server_key_iv: "2c7f3c19d5c2681ad2fcdb49c35735ce"
server_key_version: "encryptionkey@0"
I20260925 00:53:54.953099 26071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:54.971616 26071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:54.974836 26071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:54.975032 26071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:54.975647 26071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:54.976513 26071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:54.976560 26071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.976614 26071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:54.976644 26071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:54.985723 26071 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:46675
I20260925 00:53:54.985755 26199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:46675 every 8 connection(s)
I20260925 00:53:54.986812 26071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:53:54.989835 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26071
I20260925 00:53:54.990348 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:35173
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:40467
--webserver_port=44551
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:54 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:53:55.004459 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:54.988760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:38183 (local address 127.16.202.62:40467)
0925 00:53:54.988887 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:54.988891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:54.990210 (+ 1319us) server_negotiation.cc:407] Connection header received
0925 00:53:54.991416 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:54.991419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:54.991476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:54.991554 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:54.992902 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.993675 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.995011 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.995366 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.995481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.004310 (+ 8829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:38183: BlockingRecv error: recv got EOF from 127.16.202.2:38183 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:53:55.005093 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:54.989210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:38183)
0925 00:53:54.990015 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:54.990081 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:54.991156 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:54.991791 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:54.991802 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:54.992189 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:54.992737 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.992756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.993827 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.993830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.994870 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:54.994881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:54.995513 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:54.995516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:54.995663 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.001317 (+ 5654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.001338 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.004112 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":129,"threads_started":1}
W20260925 00:53:55.005705 26200 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:40467 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.202.62:40467: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:55.011387 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.006206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:56963)
0925 00:53:55.006368 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:55.006383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:55.006471 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:55.006696 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:55.006699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:55.006872 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:55.006969 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.006975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.007688 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.007692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.008255 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.008264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.008718 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.008722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.008812 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.009332 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.009345 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.011242 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 00:53:55.011520 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:55.006328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:56963 (local address 127.16.202.62:40467)
0925 00:53:55.006449 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.006453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.006468 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:55.006562 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.006566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.006604 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.006668 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.007082 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.007591 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.008386 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.008600 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.008709 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.011409 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:56963: BlockingRecv error: recv got EOF from 127.16.202.2:56963 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:55.017282 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.012074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:51929)
0925 00:53:55.012182 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:55.012196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:55.012266 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:55.012539 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:55.012541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:55.012658 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:55.012766 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.012771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.013723 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.013726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.014266 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.014274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.014854 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.014857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.014920 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.015374 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.015386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.017177 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 00:53:55.017488 26200 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:53:55.017676 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:55.012162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:51929 (local address 127.16.202.62:40467)
0925 00:53:55.012280 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.012283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.012298 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:55.012399 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.012402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.012446 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.012526 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.012863 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.013609 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.014411 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.014745 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.014841 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.017353 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:51929: BlockingRecv error: recv got EOF from 127.16.202.2:51929 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 00:53:55.106766 26203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:53:55.107060 26203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:53:55.107105 26203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:53:55.111203 26203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:53:55.111279 26203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:53:55.111340 26203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:53:55.117362 26203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/krb5kdc/oryx_127.16.202.3.keytab
--principal=oryx/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:35173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=44551
--tserver_master_addrs=127.16.202.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:53:55.118713 26203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:53:55.119988 26203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:53:55.132171 26208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:53:55.132190 26209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:55.132375 26203 server_base.cc:1034] running on GCE node
W20260925 00:53:55.132643 26211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:53:55.133215 26203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:53:55.134120 26203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:53:55.135322 26203 hybrid_clock.cc:648] HybridClock initialized: now 1790297635135244 us; error 94 us; skew 500 ppm
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297635, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:53:55.138943 26203 init.cc:377] Logged in from keytab as oryx/127.16.202.3@KRBTEST.COM (short username oryx)
I20260925 00:53:55.139911 26203 webserver.cc:460] Webserver started at http://127.16.202.3:44551/ using document root <none> and password file <none>
I20260925 00:53:55.140528 26203 fs_manager.cc:359] Metadata directory not provided
I20260925 00:53:55.140642 26203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:53:55.146396 26203 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 00:53:55.153272 26218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:55.155107 26203 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260925 00:53:55.155268 26203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "7a7693c64d9d43288a2acfebebacf405"
format_stamp: "Formatted at 2026-09-25 00:53:54 on dist-test-slave-rpk1"
server_key: "cef1daf6ac6370ccf904aa46971c279e"
server_key_iv: "fc3fccf7279d8412231b3e18a83ce684"
server_key_version: "encryptionkey@0"
I20260925 00:53:55.155882 26203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:53:55.171195 26203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:53:55.174255 26203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:53:55.174441 26203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:53:55.175027 26203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:53:55.175853 26203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:53:55.175899 26203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:55.175951 26203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:53:55.175979 26203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:53:55.184531 26203 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:35173
I20260925 00:53:55.184581 26331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:35173 every 8 connection(s)
I20260925 00:53:55.185606 26203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:53:55.187683 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26203
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:53:55.197568 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:55.187507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:33735 (local address 127.16.202.62:40467)
0925 00:53:55.187681 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.187687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.188928 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 00:53:55.189853 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.189856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.189909 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.189982 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.191448 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.192077 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.193428 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.193764 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.193850 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.197441 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:33735: BlockingRecv error: recv got EOF from 127.16.202.3:33735 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 00:53:55.198261 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.187906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:33735)
0925 00:53:55.188719 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:55.188801 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:55.189637 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:55.190187 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:55.190197 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:55.190678 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:55.191294 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.191309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.192205 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.192209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.193293 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.193306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.193925 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.193929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.194140 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.194734 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.194755 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.197315 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":134,"threads_started":1}
I20260925 00:53:55.198297 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.188860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.16.202.62:40467)
0925 00:53:55.189061 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.189064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.189072 (+ 8us) server_negotiation.cc:407] Connection header received
0925 00:53:55.189106 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.189108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.189148 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.189223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.189996 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.190808 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.191891 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.192130 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.192241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.193797 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:53:55.193803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:53:55.193822 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:53:55.193858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:53:55.195970 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:55.196426 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:55.196433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:55.196439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:55.196622 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:53:55.196930 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:53:55.196933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:53:55.196935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:53:55.197316 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:53:55.197499 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 00:53:55.197882 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 00:53:55.198140 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
W20260925 00:53:55.198735 26332 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:40467 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.202.62:40467: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:55.204064 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.199130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:49671)
0925 00:53:55.199337 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:55.199351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:55.199433 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:55.199642 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:55.199645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:55.199797 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:55.199890 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.199896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.200737 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.200740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.201291 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.201300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.201710 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.201714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.201811 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.202252 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.202263 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.203958 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147}
W20260925 00:53:55.204222 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:55.199228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:49671 (local address 127.16.202.62:40467)
0925 00:53:55.199334 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.199338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.199391 (+ 53us) server_negotiation.cc:407] Connection header received
0925 00:53:55.199511 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.199514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.199554 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.199613 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.199991 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.200618 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.201397 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.201618 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.201700 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.204109 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:49671: BlockingRecv error: recv got EOF from 127.16.202.3:49671 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:55.209285 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.204744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:39621)
0925 00:53:55.204849 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:55.204879 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:55.204956 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:55.205180 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:55.205183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:55.205317 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:55.205395 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.205401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.206073 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.206076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.206582 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.206589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.206984 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.206987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.207060 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.207533 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.207543 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.209184 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 00:53:55.209446 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:55.204810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:39621 (local address 127.16.202.62:40467)
0925 00:53:55.204948 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.204952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.204964 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:53:55.205053 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.205055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.205094 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.205155 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.205500 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.205979 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.206684 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.206892 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.206974 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.209304 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:39621: BlockingRecv error: recv got EOF from 127.16.202.3:39621 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 00:53:55.209549 26332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:55 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:55.831276 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:55.826061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:35391)
0925 00:53:55.826313 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:55.826326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:55.826414 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:55.826660 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:55.826663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:55.826834 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:55.826947 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.826954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.827766 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.827770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.828389 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:55.828401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.828860 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.828863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:55.828974 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.829418 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:55.829429 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:55.831156 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260925 00:53:55.831508 26338 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:55.826200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:35391 (local address 127.16.202.62:40467)
0925 00:53:55.826461 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:55.826465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:55.826477 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:53:55.826512 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:55.826514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:55.826556 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:55.826637 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:55.827076 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.827628 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.828505 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:55.828767 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:55.828838 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:55.831363 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:35391: BlockingRecv error: recv got EOF from 127.16.202.1:35391 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
Sep 25 00:53:56 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:56 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:56.023517 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:56.018137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:49731)
0925 00:53:56.018460 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:56.018475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:56.018590 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:56.018915 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:56.018918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:56.019109 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:56.019207 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:56.019213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.019968 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.019971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:56.020540 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:56.020549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.021017 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.021019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:56.021080 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:56.021584 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:56.021600 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:56.023386 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":170,"threads_started":1}
W20260925 00:53:56.023665 26338 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:56.018223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:49731 (local address 127.16.202.62:40467)
0925 00:53:56.018376 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:56.018380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:56.018551 (+ 171us) server_negotiation.cc:407] Connection header received
0925 00:53:56.018696 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:56.018699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:56.018757 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:56.018831 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:56.019326 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.019871 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.020650 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.020914 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.020993 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:56.023537 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:49731: BlockingRecv error: recv got EOF from 127.16.202.2:49731 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 00:53:56 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:56 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:56.215381 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:56.210187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:50203)
0925 00:53:56.210446 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:56.210460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:56.210579 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:56.210817 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:56.210820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:56.211019 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:56.211117 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:56.211125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.211882 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.211886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:56.212442 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:56.212451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.212953 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.212965 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:56.213047 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:56.213512 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:56.213522 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:56.215218 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
W20260925 00:53:56.215497 26338 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:56.210244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:50203 (local address 127.16.202.62:40467)
0925 00:53:56.210378 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:56.210383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:56.210548 (+ 165us) server_negotiation.cc:407] Connection header received
0925 00:53:56.210664 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:56.210667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:56.210719 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:56.210782 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:56.211240 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.211778 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.212558 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.212841 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.212928 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:56.215379 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:50203: BlockingRecv error: recv got EOF from 127.16.202.3:50203 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 00:53:56 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:56 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:56.837984 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:56.832184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:35529)
0925 00:53:56.832524 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:56.832541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:56.832646 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:56.832950 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:56.832953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:56.833141 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:56.833238 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:56.833245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.834070 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.834074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:56.834749 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:56.834761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.835246 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.835249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:56.835361 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:56.835855 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:56.835867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:56.837826 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":97,"threads_started":1}
W20260925 00:53:56.838141 26342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:56.832322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:35529 (local address 127.16.202.62:40467)
0925 00:53:56.832637 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:56.832642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:56.832662 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:53:56.832763 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:56.832766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:56.832815 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:56.832904 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:56.833340 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.833954 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.834902 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:56.835127 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:56.835210 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:56.838004 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:35529: BlockingRecv error: recv got EOF from 127.16.202.1:35529 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":112,"threads_started":1}
Sep 25 00:53:57 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:57 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:57.029870 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:57.024303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:54949)
0925 00:53:57.024621 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:57.024637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:57.024814 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:57.025084 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:57.025087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:57.025275 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:57.025368 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:57.025373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.026184 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.026188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:57.026745 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:57.026753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.027250 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.027253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:57.027335 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:57.027833 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:57.027845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:57.029694 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":75,"threads_started":1}
W20260925 00:53:57.029986 26342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:57.024437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:54949 (local address 127.16.202.62:40467)
0925 00:53:57.024663 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:57.024667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:57.024752 (+ 85us) server_negotiation.cc:407] Connection header received
0925 00:53:57.024912 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:57.024916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:57.024973 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:57.025050 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:57.025482 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.026079 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.026859 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.027141 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.027233 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:57.029853 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:54949: BlockingRecv error: recv got EOF from 127.16.202.2:54949 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 25 00:53:57 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:57 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:57.221808 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:57.216258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:39163)
0925 00:53:57.216652 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:57.216710 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:57.216851 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:57.217136 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:57.217138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:57.217311 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:57.217423 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:57.217429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.218201 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.218205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:57.218761 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:57.218769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.219254 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.219258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:57.219328 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:57.219802 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:57.219813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:57.221666 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
W20260925 00:53:57.221946 26342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:57.216421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:39163 (local address 127.16.202.62:40467)
0925 00:53:57.216568 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:57.216574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:57.216813 (+ 239us) server_negotiation.cc:407] Connection header received
0925 00:53:57.216950 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:57.216954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:57.217010 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:57.217098 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:57.217547 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.218089 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.218882 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.219123 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.219211 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:57.221813 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:39163: BlockingRecv error: recv got EOF from 127.16.202.3:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 00:53:57 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:57 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:57.844134 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:57.838796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:40539)
0925 00:53:57.839108 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:57.839123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:57.839219 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:57.839450 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:57.839452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:57.839624 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:57.839719 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:57.839724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.840490 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.840494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:57.841134 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:57.841146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.841608 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.841618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:57.841694 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:57.842195 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:57.842206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:57.844003 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
W20260925 00:53:57.844276 26346 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:57.838922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:40539 (local address 127.16.202.62:40467)
0925 00:53:57.839163 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:57.839167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:57.839182 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:57.839299 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:57.839301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:57.839342 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:57.839416 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:57.839809 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.840349 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.841249 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:57.841477 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:57.841554 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:57.844154 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:40539: BlockingRecv error: recv got EOF from 127.16.202.1:40539 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
Sep 25 00:53:58 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:58 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:58.035842 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:58.030633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:59221)
0925 00:53:58.030897 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:58.030913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:58.031010 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:58.031215 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:58.031217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:58.031390 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:58.031524 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:58.031529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.032273 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.032276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:58.032897 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:58.032907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.033259 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.033262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:58.033318 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:58.033796 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:58.033807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:58.035700 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
W20260925 00:53:58.035945 26346 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:58.030719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:59221 (local address 127.16.202.62:40467)
0925 00:53:58.030862 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:58.030867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:58.030977 (+ 110us) server_negotiation.cc:407] Connection header received
0925 00:53:58.031081 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:58.031083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:58.031126 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:58.031190 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:58.031627 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.032174 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.032988 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.033187 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.033251 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:58.035845 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:59221: BlockingRecv error: recv got EOF from 127.16.202.2:59221 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 00:53:58 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:58 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:58.228240 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:58.222602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:45511)
0925 00:53:58.222871 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:58.222884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:58.223006 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:58.223293 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:58.223297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:58.223513 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:58.223625 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:58.223631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.224442 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.224446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:58.225205 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:58.225217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.225736 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.225739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:58.225799 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:58.226317 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:58.226328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:58.228120 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
W20260925 00:53:58.228418 26346 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:58.222715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:45511 (local address 127.16.202.62:40467)
0925 00:53:58.222867 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:58.222871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:58.223004 (+ 133us) server_negotiation.cc:407] Connection header received
0925 00:53:58.223099 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:58.223102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:58.223156 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:58.223227 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:58.223778 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.224331 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.225395 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.225633 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.225728 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:58.228296 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:45511: BlockingRecv error: recv got EOF from 127.16.202.3:45511 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 00:53:58 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:58 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:58.850771 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:58.845058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:49295)
0925 00:53:58.845314 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:58.845327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:58.845417 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:58.845663 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:58.845665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:58.845857 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:58.845956 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:58.845962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.846829 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.846832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:58.847407 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:58.847417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.847945 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.847949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:58.848021 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:58.848584 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:58.848596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:58.850625 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20260925 00:53:58.850967 26350 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:58.845199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:49295 (local address 127.16.202.62:40467)
0925 00:53:58.845454 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:58.845458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:58.845471 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:53:58.845513 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:58.845515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:58.845555 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:58.845640 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:58.846087 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.846681 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.847547 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:58.847792 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:58.847880 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:58.850824 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:49295: BlockingRecv error: recv got EOF from 127.16.202.1:49295 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
Sep 25 00:53:59 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:59 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:59.042019 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:59.036631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:52691)
0925 00:53:59.036896 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:59.036909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:59.037023 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:59.037278 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:59.037280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:59.037462 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:59.037559 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:59.037564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.038315 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.038319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:59.038873 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:59.038882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.039363 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.039367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:59.039439 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:59.039913 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:59.039926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:59.041877 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
W20260925 00:53:59.042135 26350 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:59.036727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:52691 (local address 127.16.202.62:40467)
0925 00:53:59.036841 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:59.036844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:59.036969 (+ 125us) server_negotiation.cc:407] Connection header received
0925 00:53:59.037123 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:59.037126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:59.037179 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:59.037250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:59.037661 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.038196 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.038995 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.039234 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.039320 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:59.042019 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:52691: BlockingRecv error: recv got EOF from 127.16.202.2:52691 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 00:53:59 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:59 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:59.234681 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:59.229115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:57831)
0925 00:53:59.229384 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:59.229396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:59.229496 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:59.229720 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:59.229722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:59.229900 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:59.229999 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:59.230004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.230712 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.230716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:59.231278 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:59.231287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.231828 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.231832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:59.231938 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:59.232590 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:59.232605 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:59.234542 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
W20260925 00:53:59.234779 26350 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:59.229208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:57831 (local address 127.16.202.62:40467)
0925 00:53:59.229326 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:59.229330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:59.229459 (+ 129us) server_negotiation.cc:407] Connection header received
0925 00:53:59.229573 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:59.229575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:59.229620 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:59.229693 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:59.230087 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.230600 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.231418 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.231685 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.231783 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:59.234679 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:57831: BlockingRecv error: recv got EOF from 127.16.202.3:57831 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 00:53:59 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:53:59 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:53:59.857147 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:53:59.851649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:33583)
0925 00:53:59.851954 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 00:53:59.851970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:53:59.852061 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:53:59.852305 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:53:59.852307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:53:59.852512 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:53:59.852608 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:59.852614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.853462 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.853465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:59.854033 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:53:59.854042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.854519 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.854522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:53:59.854598 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:59.855127 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:53:59.855139 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:53:59.856991 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":76,"threads_started":1}
W20260925 00:53:59.857252 26354 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:53:59.851778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:33583 (local address 127.16.202.62:40467)
0925 00:53:59.852022 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 00:53:59.852025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:53:59.852040 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:53:59.852140 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:53:59.852142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:53:59.852183 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:53:59.852256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:53:59.852764 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.853326 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.854158 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:53:59.854384 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:53:59.854485 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:53:59.857131 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:33583: BlockingRecv error: recv got EOF from 127.16.202.1:33583 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
Sep 25 00:54:00 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:00 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:00.048852 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:00.042934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:44389)
0925 00:54:00.043245 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:00.043262 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:00.043406 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:00.043667 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:00.043669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:00.043869 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:00.043981 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:00.043987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.044753 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.044757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:00.045333 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:00.045340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.045777 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.045781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:00.045862 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:00.046363 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:00.046375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:00.048664 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
W20260925 00:54:00.048975 26354 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:00.043122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:44389 (local address 127.16.202.62:40467)
0925 00:54:00.043291 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:00.043295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:00.043314 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:54:00.043514 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:00.043517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:00.043561 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:00.043628 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:00.044104 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.044624 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.045447 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.045670 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.045761 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:00.048862 (+ 3101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:44389: BlockingRecv error: recv got EOF from 127.16.202.2:44389 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 00:54:00 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:00 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:00.244040 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:00.235398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:37971)
0925 00:54:00.235689 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:00.235701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:00.235791 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:00.236100 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:00.236104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:00.236285 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:00.236385 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:00.236391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.237180 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.237183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:00.237742 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:00.237751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.238173 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.238176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:00.238239 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:00.241739 (+ 3500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:00.241753 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:00.243857 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
W20260925 00:54:00.244171 26354 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:00.235473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:37971 (local address 127.16.202.62:40467)
0925 00:54:00.235599 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:00.235603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:00.235762 (+ 159us) server_negotiation.cc:407] Connection header received
0925 00:54:00.235856 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:00.235858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:00.235904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:00.236007 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:00.236503 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.237066 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.237850 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.238077 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.238153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:00.244059 (+ 5906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:37971: BlockingRecv error: recv got EOF from 127.16.202.3:37971 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 00:54:00 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:00 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:00.863777 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:00.858241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:34137)
0925 00:54:00.858502 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:00.858518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:00.858625 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:00.858888 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:00.858890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:00.859118 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:00.859231 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:00.859238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.860143 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.860148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:00.860759 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:00.860770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.861226 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.861230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:00.861310 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:00.861784 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:00.861795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:00.863661 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
W20260925 00:54:00.863962 26358 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:00.858366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34137 (local address 127.16.202.62:40467)
0925 00:54:00.858608 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:00.858612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:00.858629 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:54:00.858707 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:00.858710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:00.858762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:00.858849 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:00.859342 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.860002 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.860874 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:00.861127 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:00.861213 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:00.863810 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:34137: BlockingRecv error: recv got EOF from 127.16.202.1:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
Sep 25 00:54:01 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:01 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:01.055851 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:01.049937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:38873)
0925 00:54:01.050283 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:01.050298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:01.050411 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:01.050670 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:01.050672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:01.050911 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:01.051035 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:01.051042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.052004 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.052008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:01.052568 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:01.052576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.053057 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.053061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:01.053143 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:01.053695 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:01.053707 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:01.055692 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":94,"threads_started":1}
W20260925 00:54:01.055981 26358 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:01.049985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:38873 (local address 127.16.202.62:40467)
0925 00:54:01.050124 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:01.050127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:01.050359 (+ 232us) server_negotiation.cc:407] Connection header received
0925 00:54:01.050507 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:01.050510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:01.050561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:01.050625 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:01.051140 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.051891 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.052713 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.052949 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.053040 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:01.055874 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:38873: BlockingRecv error: recv got EOF from 127.16.202.2:38873 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 00:54:01 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:01 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:01.250540 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:01.245064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:55249)
0925 00:54:01.245337 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:01.245350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:01.245459 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:01.245710 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:01.245712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:01.245891 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:01.245998 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:01.246004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.246869 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.246873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:01.247459 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:01.247469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.247918 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.247921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:01.247984 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:01.248534 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:01.248547 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:01.250412 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
W20260925 00:54:01.250720 26358 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:01.245167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:55249 (local address 127.16.202.62:40467)
0925 00:54:01.245338 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:01.245342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:01.245482 (+ 140us) server_negotiation.cc:407] Connection header received
0925 00:54:01.245547 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:01.245550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:01.245611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:01.245685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:01.246134 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.246727 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.247585 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.247821 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.247905 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:01.250596 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:55249: BlockingRecv error: recv got EOF from 127.16.202.3:55249 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 25 00:54:01 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:01 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:01.870633 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:01.864662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:44787)
0925 00:54:01.865066 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:01.865080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:01.865177 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:01.865512 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:01.865515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:01.865712 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:01.865849 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:01.865858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.866739 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.866743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:01.867329 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:01.867339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.867830 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.867834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:01.867918 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:01.868400 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:01.868418 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:01.870475 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":86,"threads_started":1}
W20260925 00:54:01.870824 26362 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:01.864881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:44787 (local address 127.16.202.62:40467)
0925 00:54:01.865110 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:01.865114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:01.865130 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:54:01.865325 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:01.865327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:01.865375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:01.865448 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:01.865977 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.866589 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.867474 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:01.867704 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:01.867795 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:01.870704 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:44787: BlockingRecv error: recv got EOF from 127.16.202.1:44787 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":62,"threads_started":1}
Sep 25 00:54:02 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:02 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:02.063036 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:02.056750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:47591)
0925 00:54:02.057133 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:02.057147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:02.057264 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:02.057533 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:02.057536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:02.057738 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:02.057833 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:02.057850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.058609 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.058612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:02.059186 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:02.059196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.059705 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.059709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:02.059793 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:02.060649 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:02.060663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:02.062866 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":85,"threads_started":1}
W20260925 00:54:02.063210 26362 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:02.056914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:47591 (local address 127.16.202.62:40467)
0925 00:54:02.057084 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:02.057089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:02.057223 (+ 134us) server_negotiation.cc:407] Connection header received
0925 00:54:02.057355 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:02.057358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:02.057412 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:02.057486 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:02.057969 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.058492 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.059306 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.059590 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.059682 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:02.063098 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:47591: BlockingRecv error: recv got EOF from 127.16.202.2:47591 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 00:54:02 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:02 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:02.257344 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:02.251465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:32901)
0925 00:54:02.251800 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:02.251815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:02.251938 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:02.252236 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:02.252240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:02.252485 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:02.252605 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:02.252613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.253530 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.253534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:02.254131 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:02.254141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.254668 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.254671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:02.254737 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:02.255319 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:02.255333 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:02.257214 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":73,"threads_started":1}
W20260925 00:54:02.257560 26362 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:02.251600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:32901 (local address 127.16.202.62:40467)
0925 00:54:02.251752 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:02.251762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:02.251903 (+ 141us) server_negotiation.cc:407] Connection header received
0925 00:54:02.252031 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:02.252034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:02.252094 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:02.252182 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:02.252758 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.253399 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.254267 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.254535 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.254633 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:02.257377 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:32901: BlockingRecv error: recv got EOF from 127.16.202.3:32901 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 00:54:02 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:02 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:02.877149 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:02.871500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:51873)
0925 00:54:02.871789 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:02.871807 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:02.871918 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:02.872200 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:02.872203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:02.872387 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:02.872485 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:02.872491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.873339 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.873342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:02.873969 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:02.873979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.874585 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.874590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:02.874679 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:02.875243 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:02.875255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:02.877030 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
W20260925 00:54:02.877381 26366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:02.871671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:51873 (local address 127.16.202.62:40467)
0925 00:54:02.871967 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:02.871971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:02.871985 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:54:02.872025 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:02.872027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:02.872068 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:02.872149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:02.872605 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.873198 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.874099 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:02.874435 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:02.874544 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:02.877189 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:51873: BlockingRecv error: recv got EOF from 127.16.202.1:51873 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
Sep 25 00:54:03 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:03 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:03.070122 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:03.064013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:58367)
0925 00:54:03.064374 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:03.064389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:03.064524 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:03.064850 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:03.064853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:03.065088 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:03.065193 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:03.065200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.066205 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.066210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:03.066782 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:03.066792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.067388 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.067392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:03.067484 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:03.068106 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:03.068122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:03.070004 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":86,"threads_started":1}
W20260925 00:54:03.070289 26366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:03.064127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:58367 (local address 127.16.202.62:40467)
0925 00:54:03.064320 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:03.064326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:03.064483 (+ 157us) server_negotiation.cc:407] Connection header received
0925 00:54:03.064632 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:03.064636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:03.064717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:03.064804 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:03.065352 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.066074 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.066933 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.067244 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.067356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:03.070163 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:58367: BlockingRecv error: recv got EOF from 127.16.202.2:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 00:54:03 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:03 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:03.264029 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:03.258347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:51945)
0925 00:54:03.258609 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:03.258622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:03.258725 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:03.259074 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:03.259076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:03.259292 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:03.259438 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:03.259449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.260323 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.260327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:03.261052 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:03.261065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.261499 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.261502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:03.261573 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:03.262038 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:03.262050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:03.263902 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
W20260925 00:54:03.264199 26366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:03.258489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:51945 (local address 127.16.202.62:40467)
0925 00:54:03.258622 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:03.258626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:03.258677 (+ 51us) server_negotiation.cc:407] Connection header received
0925 00:54:03.258866 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:03.258868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:03.258921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:03.258993 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:03.259571 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.260225 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.261159 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.261375 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.261487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:03.264076 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:51945: BlockingRecv error: recv got EOF from 127.16.202.3:51945 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 00:54:03 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:03 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:03.883275 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:03.877932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:57109)
0925 00:54:03.878240 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:03.878254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:03.878355 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:03.878603 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:03.878606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:03.878776 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:03.878872 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:03.878878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.879604 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.879607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:03.880162 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:03.880170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.880604 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.880607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:03.880720 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:03.881239 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:03.881251 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:03.883155 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":74,"threads_started":1}
W20260925 00:54:03.883471 26370 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:03.878039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:57109 (local address 127.16.202.62:40467)
0925 00:54:03.878284 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:03.878287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:03.878300 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:54:03.878433 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:03.878435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:03.878483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:03.878555 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:03.878961 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.879511 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.880250 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:03.880483 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:03.880552 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:03.883339 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:57109: BlockingRecv error: recv got EOF from 127.16.202.1:57109 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
Sep 25 00:54:04 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:04 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:04.076445 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:04.070836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:42939)
0925 00:54:04.071113 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:04.071127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:04.071238 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:04.071487 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:04.071490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:04.071692 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:04.071793 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:04.071798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.072555 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.072559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:04.073141 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:04.073149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.073591 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.073594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:04.073674 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:04.074166 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:04.074180 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:04.076277 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
W20260925 00:54:04.076642 26370 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:04.070942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:42939 (local address 127.16.202.62:40467)
0925 00:54:04.071083 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:04.071087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:04.071196 (+ 109us) server_negotiation.cc:407] Connection header received
0925 00:54:04.071330 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:04.071333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:04.071383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:04.071447 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:04.071906 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.072439 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.073255 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.073487 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.073574 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:04.076525 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:42939: BlockingRecv error: recv got EOF from 127.16.202.2:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 00:54:04 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:04 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:04.270440 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:04.264940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:56859)
0925 00:54:04.265254 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:04.265267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:04.265367 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:04.265593 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:04.265596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:04.265779 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:04.265885 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:04.265891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.266759 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.266762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:04.267343 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:04.267352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.267781 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.267784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:04.267843 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:04.268283 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:04.268294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:04.270325 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
W20260925 00:54:04.270526 26370 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:04.265114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:56859 (local address 127.16.202.62:40467)
0925 00:54:04.265256 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:04.265260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:04.265338 (+ 78us) server_negotiation.cc:407] Connection header received
0925 00:54:04.265442 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:04.265444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:04.265490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:04.265564 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:04.266048 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.266639 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.267461 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.267688 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.267765 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:04.270435 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:56859: BlockingRecv error: recv got EOF from 127.16.202.3:56859 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 00:54:04 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:04 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:04.889451 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:04.884124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:59005)
0925 00:54:04.884399 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:04.884412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:04.884495 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:04.884725 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:04.884728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:04.884898 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:04.885022 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:04.885028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.885717 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.885719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:04.886278 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:04.886285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.886699 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.886701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:04.886761 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:04.887231 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:04.887241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:04.889260 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
W20260925 00:54:04.889555 26374 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:04.884225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:59005 (local address 127.16.202.62:40467)
0925 00:54:04.884490 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:04.884494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:04.884507 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:54:04.884559 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:04.884561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:04.884599 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:04.884699 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:04.885119 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.885640 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.886386 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:04.886608 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:04.886684 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:04.889421 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:59005: BlockingRecv error: recv got EOF from 127.16.202.1:59005 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":116,"threads_started":1}
Sep 25 00:54:05 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:05 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:05.083312 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:05.077442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:58581)
0925 00:54:05.077726 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:05.077774 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:05.077902 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:05.078154 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:05.078156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:05.078365 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:05.078472 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:05.078479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.079318 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.079321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:05.079920 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:05.079930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.080490 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.080493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:05.080565 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:05.081228 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:05.081243 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:05.083160 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
W20260925 00:54:05.083407 26374 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:05.077545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:58581 (local address 127.16.202.62:40467)
0925 00:54:05.077728 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:05.077732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:05.077842 (+ 110us) server_negotiation.cc:407] Connection header received
0925 00:54:05.077999 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:05.078001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:05.078044 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:05.078119 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:05.078620 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.079193 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.080059 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.080343 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.080444 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:05.083305 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:58581: BlockingRecv error: recv got EOF from 127.16.202.2:58581 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 00:54:05 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:05 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:05.280746 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:05.271478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:32989)
0925 00:54:05.271756 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:05.271772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:05.271920 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:05.272306 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:05.272309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:05.272531 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:05.272645 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:05.272654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.273545 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.273550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:05.274119 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:05.274127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.274630 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.274633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:05.274707 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:05.278489 (+ 3782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:05.278506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:05.280515 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
W20260925 00:54:05.280829 26374 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:05.271665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:32989 (local address 127.16.202.62:40467)
0925 00:54:05.271808 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:05.271811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:05.271863 (+ 52us) server_negotiation.cc:407] Connection header received
0925 00:54:05.272107 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:05.272110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:05.272171 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:05.272252 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:05.272783 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.273431 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.274244 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.274500 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.274589 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:05.280718 (+ 6129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:32989: BlockingRecv error: recv got EOF from 127.16.202.3:32989 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 00:54:05 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:05 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:05.895916 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:05.890325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:33917)
0925 00:54:05.890619 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:05.890631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:05.890719 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:05.891067 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:05.891070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:05.891245 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:05.891399 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:05.891405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.892175 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.892178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:05.892850 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:05.892861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.893273 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.893276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:05.893337 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:05.893861 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:05.893872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:05.895796 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
W20260925 00:54:05.896091 26378 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:05.890430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:33917 (local address 127.16.202.62:40467)
0925 00:54:05.890651 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:05.890654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:05.890671 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:54:05.890873 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:05.890875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:05.890925 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:05.891008 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:05.891508 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.892056 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.892962 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:05.893182 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:05.893259 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:05.895959 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:33917: BlockingRecv error: recv got EOF from 127.16.202.1:33917 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
Sep 25 00:54:06 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:06 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:06.089560 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:06.084117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:40749)
0925 00:54:06.084426 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:06.084439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:06.084537 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:06.084783 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:06.084785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:06.084958 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:06.085067 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:06.085073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.085845 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.085849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:06.086411 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:06.086419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.086828 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.086832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:06.086898 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:06.087452 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:06.087464 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:06.089435 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
W20260925 00:54:06.089685 26378 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:06.084214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:40749 (local address 127.16.202.62:40467)
0925 00:54:06.084342 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:06.084346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:06.084501 (+ 155us) server_negotiation.cc:407] Connection header received
0925 00:54:06.084617 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:06.084619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:06.084663 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:06.084749 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:06.085171 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.085687 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.086507 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.086718 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.086787 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:06.089580 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:40749: BlockingRecv error: recv got EOF from 127.16.202.2:40749 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 00:54:06 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:06 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:06.287171 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:06.281677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:58237)
0925 00:54:06.282072 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:06.282088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:06.282207 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:06.282489 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:06.282493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:06.282695 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:06.282829 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:06.282837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.283637 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.283640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:06.284215 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:06.284224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.284745 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.284750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:06.284822 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:06.285374 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:06.285386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:06.287053 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":83,"threads_started":1}
W20260925 00:54:06.287330 26378 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:06.281773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:58237 (local address 127.16.202.62:40467)
0925 00:54:06.281904 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:06.281909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:06.282164 (+ 255us) server_negotiation.cc:407] Connection header received
0925 00:54:06.282304 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:06.282306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:06.282367 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:06.282437 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:06.282961 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.283514 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.284351 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.284591 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.284707 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:06.287212 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:58237: BlockingRecv error: recv got EOF from 127.16.202.3:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 00:54:06 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:06 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:06.902977 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:06.896867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:33963)
0925 00:54:06.897133 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:06.897147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:06.897240 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:06.897561 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:06.897564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:06.897777 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:06.897888 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:06.897895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.898744 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.898748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:06.899365 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:06.899429 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.899966 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.899969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:06.900046 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:06.900706 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:06.900721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:06.902836 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
W20260925 00:54:06.903198 26382 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:06.897018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:33963 (local address 127.16.202.62:40467)
0925 00:54:06.897239 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:06.897244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:06.897258 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:54:06.897305 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:06.897308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:06.897356 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:06.897449 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:06.898050 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.898607 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.899563 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:06.899811 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:06.899920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:06.903054 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:33963: BlockingRecv error: recv got EOF from 127.16.202.1:33963 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
Sep 25 00:54:07 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:07 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:07.095714 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:07.090434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:54085)
0925 00:54:07.090725 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:07.090738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:07.090839 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:07.091171 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:07.091174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:07.091359 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:07.091456 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:07.091462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.092248 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.092251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:07.092835 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:07.092843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.093280 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.093282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:07.093339 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:07.093842 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:07.093854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:07.095568 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
W20260925 00:54:07.095831 26382 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:07.090567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:54085 (local address 127.16.202.62:40467)
0925 00:54:07.090723 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:07.090727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:07.090796 (+ 69us) server_negotiation.cc:407] Connection header received
0925 00:54:07.090978 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:07.090981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:07.091030 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:07.091114 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:07.091592 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.092149 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.092951 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.093185 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.093267 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:07.095713 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:54085: BlockingRecv error: recv got EOF from 127.16.202.2:54085 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 00:54:07 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:07 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:07.293751 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:07.288105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:59883)
0925 00:54:07.288530 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:07.288544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:07.288654 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:07.288921 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:07.288924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:07.289143 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:07.289252 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:07.289258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.290017 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.290021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:07.290581 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:07.290589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.291041 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.291044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:07.291124 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:07.291640 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:07.291652 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:07.293595 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":80,"threads_started":1}
W20260925 00:54:07.293855 26382 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:07.288253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:59883 (local address 127.16.202.62:40467)
0925 00:54:07.288415 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:07.288418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:07.288616 (+ 198us) server_negotiation.cc:407] Connection header received
0925 00:54:07.288763 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:07.288766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:07.288818 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:07.288883 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:07.289366 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.289899 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.290696 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.290929 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.291022 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:07.293751 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:59883: BlockingRecv error: recv got EOF from 127.16.202.3:59883 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 00:54:07 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:07 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:07.909852 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:07.903906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:55113)
0925 00:54:07.904240 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:07.904254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:07.904346 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:07.904591 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:07.904593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:07.904789 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:07.904906 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:07.904912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.905789 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.905792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:07.906414 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:07.906428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.907119 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.907122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:07.907191 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:07.907721 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:07.907732 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:07.909711 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
W20260925 00:54:07.909973 26386 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:07.904038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:55113 (local address 127.16.202.62:40467)
0925 00:54:07.904284 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:07.904288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:07.904303 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:54:07.904434 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:07.904437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:07.904486 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:07.904565 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:07.905051 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.905639 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.906571 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:07.906855 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:07.906960 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:07.909856 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:55113: BlockingRecv error: recv got EOF from 127.16.202.1:55113 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":59,"threads_started":1}
Sep 25 00:54:08 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:08 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:08.102694 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:08.096931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:45181)
0925 00:54:08.097235 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:08.097250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:08.097392 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:08.097772 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:08.097774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:08.098009 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:08.098121 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:08.098128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.099081 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.099085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:08.099726 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:08.099738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.100211 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.100214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:08.100282 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:08.100841 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:08.100853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:08.102568 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
W20260925 00:54:08.102807 26386 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:08.097149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:45181 (local address 127.16.202.62:40467)
0925 00:54:08.097276 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:08.097279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:08.097299 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:54:08.097564 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:08.097568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:08.097629 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:08.097728 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:08.098242 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.098946 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.099864 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.100097 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.100185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:08.102704 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:45181: BlockingRecv error: recv got EOF from 127.16.202.2:45181 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 00:54:08 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:08 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:08.299958 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:08.294720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:46247)
0925 00:54:08.295035 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:08.295049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:08.295159 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:08.295409 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:08.295412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:08.295626 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:08.295720 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:08.295726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.296459 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.296462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:08.297063 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:08.297072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.297506 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.297511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:08.297593 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:08.298076 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:08.298089 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:08.299800 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
W20260925 00:54:08.300055 26386 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:08.294842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:46247 (local address 127.16.202.62:40467)
0925 00:54:08.294983 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:08.294988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:08.295120 (+ 132us) server_negotiation.cc:407] Connection header received
0925 00:54:08.295250 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:08.295253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:08.295303 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:08.295368 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:08.295838 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.296352 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.297177 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.297394 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.297489 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:08.299932 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:46247: BlockingRecv error: recv got EOF from 127.16.202.3:46247 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 00:54:08 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:08 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:08.915988 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:08.910578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:58497)
0925 00:54:08.910862 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:08.910875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:08.910968 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:08.911213 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:08.911216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:08.911387 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:08.911492 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:08.911497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.912256 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.912259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:08.912962 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:08.912973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.913464 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.913467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:08.913537 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:08.914006 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:08.914017 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:08.915867 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
W20260925 00:54:08.916136 26390 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:08.910701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:58497 (local address 127.16.202.62:40467)
0925 00:54:08.911025 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:08.911028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:08.911041 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:54:08.911075 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:08.911077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:08.911118 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:08.911190 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:08.911585 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.912131 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.913090 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:08.913332 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:08.913414 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:08.916010 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:58497: BlockingRecv error: recv got EOF from 127.16.202.1:58497 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
Sep 25 00:54:09 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:09 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:09.109371 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:09.103531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:38913)
0925 00:54:09.103780 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:09.103793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:09.103892 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:09.104188 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:09.104191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:09.104366 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:09.104479 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:09.104485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.105319 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.105322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:09.105914 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:09.105928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.106475 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.106477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:09.106544 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:09.107048 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:09.107061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:09.109206 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
W20260925 00:54:09.109493 26390 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:09.103620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:38913 (local address 127.16.202.62:40467)
0925 00:54:09.103791 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:09.103794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:09.103857 (+ 63us) server_negotiation.cc:407] Connection header received
0925 00:54:09.104018 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:09.104021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:09.104072 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:09.104153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:09.104585 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.105194 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.106035 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.106359 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.106456 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:09.109376 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:38913: BlockingRecv error: recv got EOF from 127.16.202.2:38913 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 00:54:09 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:09 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.3@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:09.306767 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:09.300775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.3:35021)
0925 00:54:09.301085 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:09.301099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:09.301202 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:09.301511 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:09.301514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:09.301683 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:09.301779 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:09.301784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.302832 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.302836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:09.303407 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:09.303415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.304027 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.304030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:09.304101 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:09.304634 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:09.304647 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:09.306650 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
W20260925 00:54:09.306948 26390 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:09.300886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:35021 (local address 127.16.202.62:40467)
0925 00:54:09.301018 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:09.301023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:09.301166 (+ 143us) server_negotiation.cc:407] Connection header received
0925 00:54:09.301297 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:09.301300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:09.301357 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:09.301436 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:09.301874 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.302711 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.303510 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.303884 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.303993 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:09.306820 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.3:35021: BlockingRecv error: recv got EOF from 127.16.202.3:35021 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 00:54:09 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:09 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.1@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:09.922281 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:09.916978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.1:52933)
0925 00:54:09.917230 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:09.917245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:09.917351 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:09.917628 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:09.917631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:09.917801 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:09.917893 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:09.917898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.918704 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.918707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:09.919252 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:09.919258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.919706 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.919708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:09.919773 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:09.920266 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:09.920278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:09.922116 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
W20260925 00:54:09.922438 26394 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:09.917108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:52933 (local address 127.16.202.62:40467)
0925 00:54:09.917403 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:09.917407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:09.917421 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:54:09.917466 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:09.917469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:09.917516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:09.917600 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:09.918029 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.918616 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.919388 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:09.919612 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:09.919695 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:09.922255 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.1:52933: BlockingRecv error: recv got EOF from 127.16.202.1:52933 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
Sep 25 00:54:10 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:54:10 dist-test-slave-rpk1 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.202.2@KRBTEST.COM for oryx/127.16.202.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:54:10.115757 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:10.110226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40467 (local address 127.16.202.2:37421)
0925 00:54:10.110484 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:10.110497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:10.110594 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:10.110798 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:10.110801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:10.110976 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:10.111065 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:10.111070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.111912 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.111915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:10.112524 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:10.112535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.113062 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.113065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:10.113151 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.113658 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:10.113670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:10.115635 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.202.62:40467: Server oryx/127.16.202.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
W20260925 00:54:10.115942 26394 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:10.110330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:37421 (local address 127.16.202.62:40467)
0925 00:54:10.110484 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:10.110489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:10.110552 (+ 63us) server_negotiation.cc:407] Connection header received
0925 00:54:10.110670 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:10.110672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:10.110714 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:10.110770 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:10.111224 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.111777 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.112658 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.112933 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.113017 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.115801 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.202.2:37421: BlockingRecv error: recv got EOF from 127.16.202.2:37421 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 00:54:10.191429 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25919
I20260925 00:54:10.196624 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26071
I20260925 00:54:10.201457 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26203
I20260925 00:54:10.206115 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 25849
2026-09-25T00:54:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17911 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T00:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:54:10Z Disabled control of system clock
I20260925 00:54:10.234496 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:41849
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:41849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 00:54:10.347556 26399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:10.347852 26399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:10.347887 26399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:10.352015 26399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:54:10.352085 26399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:10.352099 26399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:10.352115 26399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:54:10.352129 26399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:54:10.357002 26399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:41849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:41849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:10.358105 26399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:10.359520 26399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:10.368062 26404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.368065 26407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.368062 26405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:10.368304 26399 server_base.cc:1034] running on GCE node
I20260925 00:54:10.368959 26399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:10.370127 26399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:10.371297 26399 hybrid_clock.cc:648] HybridClock initialized: now 1790297650371275 us; error 34 us; skew 500 ppm
I20260925 00:54:10.376392 26399 webserver.cc:460] Webserver started at http://127.16.202.62:38089/ using document root <none> and password file <none>
I20260925 00:54:10.377139 26399 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:10.377229 26399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:10.377460 26399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:10.379679 26399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "ad8236501b6e48acabb2851a6f367707"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "b86c0925bb149533add5d095b7235f8f"
server_key_iv: "0318c8306aa7d069e3199b62eca1e157"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.380259 26399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "ad8236501b6e48acabb2851a6f367707"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "b86c0925bb149533add5d095b7235f8f"
server_key_iv: "0318c8306aa7d069e3199b62eca1e157"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.384123 26399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:54:10.386838 26413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.388155 26399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:54:10.388281 26399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "ad8236501b6e48acabb2851a6f367707"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "b86c0925bb149533add5d095b7235f8f"
server_key_iv: "0318c8306aa7d069e3199b62eca1e157"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.388396 26399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:10.423235 26399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:10.423970 26399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:10.424151 26399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:10.431164 26399 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:41849
I20260925 00:54:10.431213 26465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:41849 every 8 connection(s)
I20260925 00:54:10.432179 26399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:54:10.436410 26466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:10.440802 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26399
I20260925 00:54:10.440920 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:54:10.441242 17192 external_mini_cluster.cc:1428] Setting key 9246230f913ebf1987fffabf9d0975a5
I20260925 00:54:10.442806 26466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707: Bootstrap starting.
I20260925 00:54:10.445195 26466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:10.445976 26466 log.cc:824] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:10.447784 26466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707: No bootstrap required, opened a new log
I20260925 00:54:10.450651 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:10.442914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.16.202.62:41849)
0925 00:54:10.443567 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:10.443584 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:10.443621 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:54:10.444298 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:10.444318 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:10.444782 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:10.445092 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:10.445506 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.446507 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.447353 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.447816 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.447981 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.448786 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:10.448791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:10.448804 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:10.449229 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:10.449289 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:10.449310 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:10.449394 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:10.449595 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 00:54:10.449887 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260925 00:54:10.451582 26466 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad8236501b6e48acabb2851a6f367707" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 41849 } }
I20260925 00:54:10.451807 26466 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:10.451866 26466 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8236501b6e48acabb2851a6f367707, State: Initialized, Role: FOLLOWER
I20260925 00:54:10.452337 26466 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad8236501b6e48acabb2851a6f367707" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 41849 } }
I20260925 00:54:10.452491 26466 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:54:10.452553 26466 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:54:10.452643 26466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:10.453632 26466 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad8236501b6e48acabb2851a6f367707" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 41849 } }
I20260925 00:54:10.453933 26466 leader_election.cc:302] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad8236501b6e48acabb2851a6f367707; no voters:
I20260925 00:54:10.454195 26466 leader_election.cc:288] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:54:10.454352 26471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:54:10.454629 26471 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [term 1 LEADER]: Becoming Leader. State: Replica: ad8236501b6e48acabb2851a6f367707, State: Running, Role: LEADER
I20260925 00:54:10.454943 26471 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad8236501b6e48acabb2851a6f367707" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 41849 } }
I20260925 00:54:10.455430 26466 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:54:10.456547 26473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad8236501b6e48acabb2851a6f367707. Latest consensus state: current_term: 1 leader_uuid: "ad8236501b6e48acabb2851a6f367707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad8236501b6e48acabb2851a6f367707" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 41849 } } }
I20260925 00:54:10.456689 26473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:10.457250 26472 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad8236501b6e48acabb2851a6f367707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad8236501b6e48acabb2851a6f367707" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 41849 } } }
I20260925 00:54:10.457403 26472 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707 [sys.catalog]: This master's current role is: LEADER
W20260925 00:54:10.460093 26486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:54:10.460165 26486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:54:10.460245 26480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:54:10.461032 26480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:54:10.466631 26480 catalog_manager.cc:1347] Generated new cluster ID: 9450809a169d4954a2bd44593138b443
I20260925 00:54:10.466708 26480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:54:10.475905 26480 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:54:10.477236 26480 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:54:10.486699 26480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad8236501b6e48acabb2851a6f367707: Generated new TSK 0
I20260925 00:54:10.487600 26480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:54:10.492054 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:41849
--builtin_ntp_servers=127.16.202.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 00:54:10.603080 26490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:10.603346 26490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:10.603399 26490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:10.607534 26490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:10.607601 26490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:10.607678 26490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:54:10.612483 26490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:41849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:10.613623 26490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:10.615106 26490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:10.624480 26495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.624533 26496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:10.624573 26490 server_base.cc:1034] running on GCE node
W20260925 00:54:10.624502 26498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:10.625118 26490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:10.625756 26490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:10.626945 26490 hybrid_clock.cc:648] HybridClock initialized: now 1790297650626925 us; error 35 us; skew 500 ppm
I20260925 00:54:10.628887 26490 webserver.cc:460] Webserver started at http://127.16.202.1:39795/ using document root <none> and password file <none>
I20260925 00:54:10.629455 26490 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:10.629523 26490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:10.629709 26490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:10.631723 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "da38228841df4a90987e337e7d18d7fe"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "ffb2422387d9a4e8d3f1d74af3798b87"
server_key_iv: "55ad8fb766289067a16db08a6d53d9c3"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.632234 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "da38228841df4a90987e337e7d18d7fe"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "ffb2422387d9a4e8d3f1d74af3798b87"
server_key_iv: "55ad8fb766289067a16db08a6d53d9c3"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.635980 26490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:54:10.638700 26504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.640082 26490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:54:10.640205 26490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "da38228841df4a90987e337e7d18d7fe"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "ffb2422387d9a4e8d3f1d74af3798b87"
server_key_iv: "55ad8fb766289067a16db08a6d53d9c3"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.640331 26490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:10.661968 26490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:10.662726 26490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:10.662906 26490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:10.663515 26490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:10.664520 26490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:10.664567 26490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.664620 26490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:10.664645 26490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.673017 26490 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43533
I20260925 00:54:10.673090 26617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43533 every 8 connection(s)
I20260925 00:54:10.673935 26490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:54:10.678675 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26490
I20260925 00:54:10.678776 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:54:10.679064 17192 external_mini_cluster.cc:1428] Setting key d5986809adf38ec2f9dbfd60d953a1ad
I20260925 00:54:10.680981 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:41849
--builtin_ntp_servers=127.16.202.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 00:54:10.683526 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:10.675779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:44015 (local address 127.16.202.62:41849)
0925 00:54:10.675960 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:10.675964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:10.677051 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 00:54:10.678465 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:10.678469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:10.678529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:10.678604 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:10.679649 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.680225 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.681554 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.681771 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.681859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.683038 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:10.683041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:10.683044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:10.683056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:10.683077 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:10.683085 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:10.683152 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:10.683319 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 00:54:10.683383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:54:10.684475 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:10.676182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:41849 (local address 127.16.202.1:44015)
0925 00:54:10.676895 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:10.676937 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:10.678242 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:10.678814 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:10.678824 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:10.678929 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:10.679472 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:10.679493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.680347 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.680350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:10.681433 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:10.681443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.681922 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.681927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:10.682084 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.682728 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:54:10.682730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:10.682773 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:54:10.682862 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:10.683164 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:10.683174 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:10.683180 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:54:10.683298 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 00:54:10.683601 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260925 00:54:10.685787 26618 heartbeater.cc:347] Connected to a master server at 127.16.202.62:41849
I20260925 00:54:10.686031 26618 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:10.686638 26618 heartbeater.cc:511] Master 127.16.202.62:41849 requested a full tablet report, sending...
I20260925 00:54:10.688397 26429 ts_manager.cc:194] Registered new tserver with Master: da38228841df4a90987e337e7d18d7fe (127.16.202.1:43533)
I20260925 00:54:10.690101 26429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.1:44015
W20260925 00:54:10.791680 26621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:10.791957 26621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:10.791997 26621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:10.796231 26621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:10.796299 26621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:10.796361 26621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:54:10.801118 26621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:41849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:10.802129 26621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:10.803458 26621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:10.812595 26627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.812619 26626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.812641 26629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:10.812904 26621 server_base.cc:1034] running on GCE node
I20260925 00:54:10.813366 26621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:10.814015 26621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:10.815196 26621 hybrid_clock.cc:648] HybridClock initialized: now 1790297650815174 us; error 33 us; skew 500 ppm
I20260925 00:54:10.817667 26621 webserver.cc:460] Webserver started at http://127.16.202.2:43887/ using document root <none> and password file <none>
I20260925 00:54:10.818326 26621 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:10.818414 26621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:10.818676 26621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:10.821023 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "40bf0141ddab46afab5570f1aa293e5a"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "23ecd431292be28c15dede548f771a55"
server_key_iv: "5970e80a8b02966324f69be4e66199b4"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.821601 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "40bf0141ddab46afab5570f1aa293e5a"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "23ecd431292be28c15dede548f771a55"
server_key_iv: "5970e80a8b02966324f69be4e66199b4"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.825340 26621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:54:10.828011 26635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.829303 26621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:54:10.829411 26621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "40bf0141ddab46afab5570f1aa293e5a"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "23ecd431292be28c15dede548f771a55"
server_key_iv: "5970e80a8b02966324f69be4e66199b4"
server_key_version: "encryptionkey@0"
I20260925 00:54:10.829515 26621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:10.838569 26621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:10.839244 26621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:10.839409 26621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:10.839967 26621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:10.840987 26621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:10.841039 26621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.841094 26621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:10.841118 26621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:10.849294 26621 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:32797
I20260925 00:54:10.849344 26748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:32797 every 8 connection(s)
I20260925 00:54:10.850200 26621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:54:10.857321 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26621
I20260925 00:54:10.857484 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:54:10.857846 17192 external_mini_cluster.cc:1428] Setting key 09c6fe1b0301c8a63ff4f47ea55d307f
I20260925 00:54:10.859437 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:10.851950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:36633 (local address 127.16.202.62:41849)
0925 00:54:10.852078 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:10.852082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:10.853147 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 00:54:10.854581 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:10.854585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:10.854646 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:10.854715 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:10.855661 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.856211 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.857389 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.857717 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.857829 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.858943 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:10.858947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:10.858950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:10.858962 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:10.858982 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:10.858990 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:10.859050 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:10.859244 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 00:54:10.859309 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:54:10.860188 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:41849
--builtin_ntp_servers=127.16.202.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 00:54:10.860296 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:10.852289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:41849 (local address 127.16.202.2:36633)
0925 00:54:10.853002 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:10.853043 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:10.854361 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:10.854905 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:10.854912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:10.854997 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:10.855521 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:10.855535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.856333 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.856337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:10.857269 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:10.857288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:10.857868 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:10.857872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:10.858046 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:10.858663 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:54:10.858665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:10.858723 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:54:10.858796 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:10.859058 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:10.859066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:10.859073 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:54:10.859204 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 00:54:10.859478 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260925 00:54:10.861616 26749 heartbeater.cc:347] Connected to a master server at 127.16.202.62:41849
I20260925 00:54:10.861896 26749 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:10.862560 26749 heartbeater.cc:511] Master 127.16.202.62:41849 requested a full tablet report, sending...
I20260925 00:54:10.863752 26429 ts_manager.cc:194] Registered new tserver with Master: 40bf0141ddab46afab5570f1aa293e5a (127.16.202.2:32797)
I20260925 00:54:10.864367 26429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.2:36633
W20260925 00:54:10.972239 26752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:10.972518 26752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:10.972554 26752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:10.976619 26752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:10.976706 26752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:10.976780 26752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:54:10.981616 26752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:41849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:10.982728 26752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:10.984190 26752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:10.994168 26758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.994168 26757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:10.994160 26760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:10.994560 26752 server_base.cc:1034] running on GCE node
I20260925 00:54:10.995049 26752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:10.995703 26752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:10.996860 26752 hybrid_clock.cc:648] HybridClock initialized: now 1790297650996840 us; error 32 us; skew 500 ppm
I20260925 00:54:10.998768 26752 webserver.cc:460] Webserver started at http://127.16.202.3:39745/ using document root <none> and password file <none>
I20260925 00:54:10.999351 26752 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:10.999419 26752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:10.999619 26752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:11.001788 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "cf442da99ebf45e28ed4f112fb7c2e6b"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "f9560b6a4a0d94e7e8ebfcaaecd2b0c8"
server_key_iv: "abffdbfece7b360ff7dcf91bd07c83ed"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.002362 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "cf442da99ebf45e28ed4f112fb7c2e6b"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "f9560b6a4a0d94e7e8ebfcaaecd2b0c8"
server_key_iv: "abffdbfece7b360ff7dcf91bd07c83ed"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.006263 26752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:54:11.009037 26766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.010356 26752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:54:11.010514 26752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "cf442da99ebf45e28ed4f112fb7c2e6b"
format_stamp: "Formatted at 2026-09-25 00:54:10 on dist-test-slave-rpk1"
server_key: "f9560b6a4a0d94e7e8ebfcaaecd2b0c8"
server_key_iv: "abffdbfece7b360ff7dcf91bd07c83ed"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.010624 26752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:11.026202 26752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:11.026901 26752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:11.027050 26752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:11.027637 26752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:11.028553 26752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:11.028597 26752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.028635 26752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:11.028656 26752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.036582 26752 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:34693
I20260925 00:54:11.036608 26879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:34693 every 8 connection(s)
I20260925 00:54:11.037458 26752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:54:11.046428 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.039246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:47305 (local address 127.16.202.62:41849)
0925 00:54:11.039420 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.039424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.040521 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 00:54:11.041843 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.041845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.041894 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.041967 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.042868 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:11.043384 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:11.044556 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:11.044858 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:11.044947 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:11.046013 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:11.046016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:11.046018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:11.046030 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:11.046048 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:11.046056 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:11.046111 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:11.046253 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 00:54:11.046311 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:54:11.047134 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26752
I20260925 00:54:11.047242 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:54:11.047465 26882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.039648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:41849 (local address 127.16.202.3:47305)
0925 00:54:11.040307 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:11.040349 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:11.041649 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:11.042123 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:11.042130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:11.042215 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:11.042754 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:11.042767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:11.043477 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:11.043480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:11.044406 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:11.044415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:11.045013 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:11.045017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:11.045179 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:11.045767 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:54:11.045768 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:11.045808 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:54:11.045873 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:11.046118 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:11.046123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:11.046126 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:54:11.046213 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 00:54:11.046492 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260925 00:54:11.047544 17192 external_mini_cluster.cc:1428] Setting key d37c21406027becdc2c1d680c6f89ae2
I20260925 00:54:11.048648 26880 heartbeater.cc:347] Connected to a master server at 127.16.202.62:41849
I20260925 00:54:11.048902 26880 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:11.049474 26880 heartbeater.cc:511] Master 127.16.202.62:41849 requested a full tablet report, sending...
I20260925 00:54:11.050453 26430 ts_manager.cc:194] Registered new tserver with Master: cf442da99ebf45e28ed4f112fb7c2e6b (127.16.202.3:34693)
I20260925 00:54:11.051029 26430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.3:47305
I20260925 00:54:11.061707 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 00:54:11.064520 26469 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:11.063658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.16.202.62:41849)
0925 00:54:11.063784 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.063788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.063939 (+ 151us) server_negotiation.cc:407] Connection header received
0925 00:54:11.064019 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.064021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.064065 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.064130 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.064329 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:54:11.064764 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26490
I20260925 00:54:11.069588 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26621
I20260925 00:54:11.074045 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26752
I20260925 00:54:11.078734 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26399
2026-09-25T00:54:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (871 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T00:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:54:11Z Disabled control of system clock
I20260925 00:54:11.106257 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45895
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 00:54:11.216187 26891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:11.216481 26891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:11.216519 26891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:11.220713 26891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:54:11.220799 26891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:11.220821 26891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:11.220844 26891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:54:11.220865 26891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:54:11.225718 26891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:45895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:11.226778 26891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:11.228127 26891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:11.236545 26899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:11.236582 26896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.236692 26891 server_base.cc:1034] running on GCE node
W20260925 00:54:11.236550 26897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.237480 26891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:11.238701 26891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:11.239892 26891 hybrid_clock.cc:648] HybridClock initialized: now 1790297651239848 us; error 60 us; skew 500 ppm
I20260925 00:54:11.242043 26891 webserver.cc:460] Webserver started at http://127.16.202.62:45993/ using document root <none> and password file <none>
I20260925 00:54:11.242811 26891 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:11.242899 26891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:11.243134 26891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:11.245443 26891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "266764f8b5cf442993a9b20416d67720"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "b1a9d05fbeeb30a241be1a4170637a0e"
server_key_iv: "d5113a6229bf32e04df6700a192d1263"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.246033 26891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "266764f8b5cf442993a9b20416d67720"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "b1a9d05fbeeb30a241be1a4170637a0e"
server_key_iv: "d5113a6229bf32e04df6700a192d1263"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.250168 26891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:54:11.252902 26905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.254165 26891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:54:11.254283 26891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "266764f8b5cf442993a9b20416d67720"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "b1a9d05fbeeb30a241be1a4170637a0e"
server_key_iv: "d5113a6229bf32e04df6700a192d1263"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.254392 26891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:11.259012 26891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:11.259637 26891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:11.259804 26891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:11.266678 26891 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:45895
I20260925 00:54:11.266690 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:45895 every 8 connection(s)
I20260925 00:54:11.267652 26891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:54:11.270905 26958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:11.271764 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26891
I20260925 00:54:11.271864 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:54:11.272148 17192 external_mini_cluster.cc:1428] Setting key 9b83fa7594c11a886b94306b5a495024
I20260925 00:54:11.276901 26958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720: Bootstrap starting.
I20260925 00:54:11.277606 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.273571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.16.202.62:45895)
0925 00:54:11.274217 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.274233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.274269 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:54:11.274930 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.274948 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.275299 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.275610 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.276339 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:11.276343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:11.276356 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:11.276367 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:11.276424 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:11.276439 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:11.276576 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:11.276643 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 00:54:11.276896 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":118,"threads_started":1}
I20260925 00:54:11.279089 26958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:11.279851 26958 log.cc:824] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:11.281934 26958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720: No bootstrap required, opened a new log
I20260925 00:54:11.285593 26958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266764f8b5cf442993a9b20416d67720" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45895 } }
I20260925 00:54:11.285885 26958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:11.285959 26958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266764f8b5cf442993a9b20416d67720, State: Initialized, Role: FOLLOWER
I20260925 00:54:11.286368 26958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266764f8b5cf442993a9b20416d67720" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45895 } }
I20260925 00:54:11.286501 26958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:54:11.286548 26958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:54:11.286643 26958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:11.287586 26958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266764f8b5cf442993a9b20416d67720" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45895 } }
I20260925 00:54:11.287925 26958 leader_election.cc:302] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 266764f8b5cf442993a9b20416d67720; no voters:
I20260925 00:54:11.288239 26958 leader_election.cc:288] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:54:11.288358 26963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:54:11.288619 26963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [term 1 LEADER]: Becoming Leader. State: Replica: 266764f8b5cf442993a9b20416d67720, State: Running, Role: LEADER
I20260925 00:54:11.288929 26963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266764f8b5cf442993a9b20416d67720" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45895 } }
I20260925 00:54:11.289554 26958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:54:11.290585 26965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 266764f8b5cf442993a9b20416d67720. Latest consensus state: current_term: 1 leader_uuid: "266764f8b5cf442993a9b20416d67720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266764f8b5cf442993a9b20416d67720" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45895 } } }
I20260925 00:54:11.290701 26965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:11.290934 26964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "266764f8b5cf442993a9b20416d67720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266764f8b5cf442993a9b20416d67720" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45895 } } }
I20260925 00:54:11.291031 26964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720 [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:11.291158 26972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:54:11.293900 26972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:54:11.299175 26972 catalog_manager.cc:1347] Generated new cluster ID: 3f05f60bc30f49a2a31e9adc7f3e3401
I20260925 00:54:11.299252 26972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:54:11.313853 26972 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:54:11.314221 26972 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:54:11.320297 26972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 266764f8b5cf442993a9b20416d67720: Generated new TSK 0
I20260925 00:54:11.321211 26972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:54:11.331190 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45895
--builtin_ntp_servers=127.16.202.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 00:54:11.442148 26982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:11.442416 26982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:11.442461 26982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:11.446545 26982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:11.446625 26982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:11.446719 26982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:54:11.451560 26982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:11.452647 26982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:11.454047 26982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:11.463722 26988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.463805 26982 server_base.cc:1034] running on GCE node
W20260925 00:54:11.463716 26987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:11.463716 26990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.464263 26982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:11.464920 26982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:11.466115 26982 hybrid_clock.cc:648] HybridClock initialized: now 1790297651466092 us; error 34 us; skew 500 ppm
I20260925 00:54:11.468006 26982 webserver.cc:460] Webserver started at http://127.16.202.1:40377/ using document root <none> and password file <none>
I20260925 00:54:11.468647 26982 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:11.468762 26982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:11.468978 26982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:11.471089 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "16588525a30d430ead820996bb266442"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "52c2561cdc977bd0a7ba3c2529c16d7a"
server_key_iv: "4fcb46bf95b05c5c51634fd922ab65b5"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.471647 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "16588525a30d430ead820996bb266442"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "52c2561cdc977bd0a7ba3c2529c16d7a"
server_key_iv: "4fcb46bf95b05c5c51634fd922ab65b5"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.475479 26982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 00:54:11.478166 26996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.479447 26982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:54:11.479564 26982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "16588525a30d430ead820996bb266442"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "52c2561cdc977bd0a7ba3c2529c16d7a"
server_key_iv: "4fcb46bf95b05c5c51634fd922ab65b5"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.479676 26982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:11.484341 26982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:11.485057 26982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:11.485215 26982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:11.485746 26982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:11.486651 26982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:11.486706 26982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.486756 26982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:11.486781 26982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.495270 26982 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43453
I20260925 00:54:11.495389 27109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43453 every 8 connection(s)
I20260925 00:54:11.496192 26982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:54:11.497521 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 26982
I20260925 00:54:11.497634 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:54:11.498008 17192 external_mini_cluster.cc:1428] Setting key 78e87c36f6bd51fa8d90160f03eb4750
I20260925 00:54:11.500124 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45895
--builtin_ntp_servers=127.16.202.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 00:54:11.502813 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.497993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:45345 (local address 127.16.202.62:45895)
0925 00:54:11.498118 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.498122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.499318 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 00:54:11.501077 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.501080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.501116 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.501187 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.502340 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:11.502344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:11.502347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:11.502363 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:11.502426 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:11.502438 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:11.502535 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:11.502647 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 00:54:11.502690 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:54:11.503706 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.498533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45895 (local address 127.16.202.1:45345)
0925 00:54:11.499186 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:11.499237 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:11.500850 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:11.501367 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:11.501376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:11.501497 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:11.502063 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:54:11.502065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:11.502106 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:54:11.502177 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:11.502531 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:11.502537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:11.502541 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:54:11.502662 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 00:54:11.502870 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
I20260925 00:54:11.504887 27110 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45895
I20260925 00:54:11.505123 27110 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:11.505515 27110 heartbeater.cc:511] Master 127.16.202.62:45895 requested a full tablet report, sending...
I20260925 00:54:11.506969 26922 ts_manager.cc:194] Registered new tserver with Master: 16588525a30d430ead820996bb266442 (127.16.202.1:43453)
W20260925 00:54:11.614521 27113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:11.614821 27113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:11.614890 27113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:11.619027 27113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:11.619097 27113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:11.619179 27113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:54:11.624080 27113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:11.625280 27113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:11.626684 27113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:11.636277 27118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:11.636267 27119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.636319 27113 server_base.cc:1034] running on GCE node
W20260925 00:54:11.636245 27121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.636847 27113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:11.637517 27113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:11.638677 27113 hybrid_clock.cc:648] HybridClock initialized: now 1790297651638660 us; error 29 us; skew 500 ppm
I20260925 00:54:11.640615 27113 webserver.cc:460] Webserver started at http://127.16.202.2:43951/ using document root <none> and password file <none>
I20260925 00:54:11.641223 27113 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:11.641294 27113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:11.641494 27113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:11.644170 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "0e15ef892a274d8faa452c237642b109"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "427020f575d589c3dff9a45bfb2ae89a"
server_key_iv: "e9d48694c7106f5ae806a43e32c217a8"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.644898 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "0e15ef892a274d8faa452c237642b109"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "427020f575d589c3dff9a45bfb2ae89a"
server_key_iv: "e9d48694c7106f5ae806a43e32c217a8"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.650041 27113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 00:54:11.652863 27127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.654297 27113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:54:11.654436 27113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "0e15ef892a274d8faa452c237642b109"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "427020f575d589c3dff9a45bfb2ae89a"
server_key_iv: "e9d48694c7106f5ae806a43e32c217a8"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.654574 27113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:11.659441 27113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:11.660079 27113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:11.660231 27113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:11.660804 27113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:11.661715 27113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:11.661758 27113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.661798 27113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:11.661826 27113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.670374 27113 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:45813
I20260925 00:54:11.670394 27240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:45813 every 8 connection(s)
I20260925 00:54:11.671316 27113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:54:11.671860 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 27113
I20260925 00:54:11.671999 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:54:11.672309 17192 external_mini_cluster.cc:1428] Setting key 685a0adf5fffa3e9f5d38e71d100c2b0
I20260925 00:54:11.674382 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45895
--builtin_ntp_servers=127.16.202.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 00:54:11.677825 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.673233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:40187 (local address 127.16.202.62:45895)
0925 00:54:11.673369 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.673373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.674608 (+ 1235us) server_negotiation.cc:407] Connection header received
0925 00:54:11.676057 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.676059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.676097 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.676168 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.677383 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:11.677386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:11.677389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:11.677402 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:11.677436 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:11.677446 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:11.677537 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:11.677647 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 00:54:11.677690 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:54:11.678646 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.673742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45895 (local address 127.16.202.2:40187)
0925 00:54:11.674456 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:11.674518 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:11.675812 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:11.676382 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:11.676391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:11.676532 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:11.677124 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:54:11.677125 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:11.677165 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:54:11.677235 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:11.677533 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:11.677538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:11.677542 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:54:11.677668 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 00:54:11.677865 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260925 00:54:11.679754 27241 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45895
I20260925 00:54:11.679972 27241 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:11.680375 27241 heartbeater.cc:511] Master 127.16.202.62:45895 requested a full tablet report, sending...
I20260925 00:54:11.681404 26922 ts_manager.cc:194] Registered new tserver with Master: 0e15ef892a274d8faa452c237642b109 (127.16.202.2:45813)
W20260925 00:54:11.785624 27244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:11.785918 27244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:11.785975 27244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:11.790194 27244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:11.790279 27244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:11.790356 27244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:54:11.795521 27244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:45895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:11.796576 27244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:11.798031 27244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:11.807644 27249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:11.807654 27250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.807799 27244 server_base.cc:1034] running on GCE node
W20260925 00:54:11.808007 27252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:11.808409 27244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:11.809118 27244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:11.810289 27244 hybrid_clock.cc:648] HybridClock initialized: now 1790297651810266 us; error 35 us; skew 500 ppm
I20260925 00:54:11.812268 27244 webserver.cc:460] Webserver started at http://127.16.202.3:43569/ using document root <none> and password file <none>
I20260925 00:54:11.812958 27244 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:11.813023 27244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:11.813211 27244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:11.815303 27244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "8ee4184d86a14573b4f7ddc41f44741d"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "ab517a3fdd338545fd0e94f8c622019c"
server_key_iv: "4ca8631ca3080159bfc7492fdfb71200"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.815850 27244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "8ee4184d86a14573b4f7ddc41f44741d"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "ab517a3fdd338545fd0e94f8c622019c"
server_key_iv: "4ca8631ca3080159bfc7492fdfb71200"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.819895 27244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:54:11.822968 27258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.824224 27244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:54:11.824328 27244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "8ee4184d86a14573b4f7ddc41f44741d"
format_stamp: "Formatted at 2026-09-25 00:54:11 on dist-test-slave-rpk1"
server_key: "ab517a3fdd338545fd0e94f8c622019c"
server_key_iv: "4ca8631ca3080159bfc7492fdfb71200"
server_key_version: "encryptionkey@0"
I20260925 00:54:11.824437 27244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:11.829195 27244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:11.829838 27244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:11.829995 27244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:11.830541 27244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:11.831480 27244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:11.831529 27244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.831568 27244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:11.831581 27244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:11.839929 27244 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:33099
I20260925 00:54:11.839963 27371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:33099 every 8 connection(s)
I20260925 00:54:11.840930 27244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:54:11.847214 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.842800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:49349 (local address 127.16.202.62:45895)
0925 00:54:11.843031 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.843036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.844130 (+ 1094us) server_negotiation.cc:407] Connection header received
0925 00:54:11.845522 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.845525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.845568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.845645 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.846745 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:11.846748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:11.846751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:54:11.846763 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:11.846784 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:54:11.846792 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:11.846874 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:11.847054 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 00:54:11.847096 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:54:11.848165 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:11.843272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:45895 (local address 127.16.202.3:49349)
0925 00:54:11.843976 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:11.844020 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:11.845251 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:11.845821 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:11.845829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:11.845929 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:11.846460 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:54:11.846461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:11.846506 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:54:11.846579 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:11.846896 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:11.846904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:11.846910 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:54:11.847076 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 00:54:11.847289 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260925 00:54:11.849407 27372 heartbeater.cc:347] Connected to a master server at 127.16.202.62:45895
I20260925 00:54:11.849660 27372 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:11.850112 27372 heartbeater.cc:511] Master 127.16.202.62:45895 requested a full tablet report, sending...
I20260925 00:54:11.850883 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 27244
I20260925 00:54:11.850986 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:54:11.851297 17192 external_mini_cluster.cc:1428] Setting key 817b5015f719af6fd724bed2ec082bb6
I20260925 00:54:11.851292 26922 ts_manager.cc:194] Registered new tserver with Master: 8ee4184d86a14573b4f7ddc41f44741d (127.16.202.3:33099)
I20260925 00:54:11.853367 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 00:54:11.856091 26961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:54:11.855218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.16.202.62:45895)
0925 00:54:11.855330 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:11.855334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:11.855515 (+ 181us) server_negotiation.cc:407] Connection header received
0925 00:54:11.855632 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:11.855635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:11.855668 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:11.855730 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:11.855966 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57986: BlockingRecv error: recv got EOF from 127.0.0.1:57986 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 00:54:11.856240 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26982
I20260925 00:54:11.861222 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 27113
I20260925 00:54:11.865585 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 27244
I20260925 00:54:11.870070 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 26891
2026-09-25T00:54:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (792 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:54:11 dist-test-slave-rpk1 krb5kdc[27382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:54:11 dist-test-slave-rpk1 krb5kdc[27382](info): set up 2 sockets
Sep 25 00:54:11 dist-test-slave-rpk1 krb5kdc[27382](info): commencing operation
krb5kdc: starting...
W20260925 00:54:12.826380 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:54:12 dist-test-slave-rpk1 krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:54:12.951521 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:35989
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:54:13.060770 27395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:13.061069 27395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:13.061110 27395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:13.065225 27395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:54:13.065295 27395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:13.065315 27395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:13.065330 27395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:54:13.065343 27395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:54:13.070739 27395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:13.071805 27395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:13.072799 27395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:13.080999 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:13.081059 27395 server_base.cc:1034] running on GCE node
W20260925 00:54:13.081018 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.081022 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:13.081769 27395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:13.082916 27395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:13.084081 27395 hybrid_clock.cc:648] HybridClock initialized: now 1790297653084065 us; error 32 us; skew 500 ppm
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:13.086870 27395 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:54:13.087838 27395 webserver.cc:460] Webserver started at http://127.16.202.62:43967/ using document root <none> and password file <none>
I20260925 00:54:13.088398 27395 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:13.088459 27395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:13.088652 27395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:13.090803 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "a88f3b20efdb4184ba214906a579253d"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "69f3d6ec795944024a90185fa1457cd0"
server_key_iv: "96c9cd6360ecad457a8a25bf16ada322"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.091379 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "a88f3b20efdb4184ba214906a579253d"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "69f3d6ec795944024a90185fa1457cd0"
server_key_iv: "96c9cd6360ecad457a8a25bf16ada322"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.095274 27395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:54:13.097858 27410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.099112 27395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:54:13.099223 27395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "a88f3b20efdb4184ba214906a579253d"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "69f3d6ec795944024a90185fa1457cd0"
server_key_iv: "96c9cd6360ecad457a8a25bf16ada322"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.099337 27395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:13.118680 27395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:13.121889 27395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:13.122071 27395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:13.128808 27395 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:35989
I20260925 00:54:13.129077 27462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:35989 every 8 connection(s)
I20260925 00:54:13.129796 27395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:54:13.132644 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:13.138188 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 27395
I20260925 00:54:13.138314 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:54:13.138330 27463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: Bootstrap starting.
I20260925 00:54:13.138615 17192 external_mini_cluster.cc:1428] Setting key 43d9fcc653736e2860ba32758b6f56fa
I20260925 00:54:13.141072 27463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:13.141808 27463 log.cc:824] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:13.143733 27463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: No bootstrap required, opened a new log
I20260925 00:54:13.147394 27463 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88f3b20efdb4184ba214906a579253d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 35989 } }
I20260925 00:54:13.147644 27463 raft_consensus.cc:374] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:13.147704 27463 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88f3b20efdb4184ba214906a579253d, State: Initialized, Role: FOLLOWER
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:54:13.148176 27463 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88f3b20efdb4184ba214906a579253d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 35989 } }
I20260925 00:54:13.148331 27463 raft_consensus.cc:388] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:54:13.148403 27463 raft_consensus.cc:482] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:54:13.148507 27463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:13.149726 27463 raft_consensus.cc:504] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88f3b20efdb4184ba214906a579253d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 35989 } }
I20260925 00:54:13.150143 27463 leader_election.cc:302] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a88f3b20efdb4184ba214906a579253d; no voters:
I20260925 00:54:13.150532 27463 leader_election.cc:288] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:54:13.150719 27468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:54:13.150957 27468 raft_consensus.cc:686] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [term 1 LEADER]: Becoming Leader. State: Replica: a88f3b20efdb4184ba214906a579253d, State: Running, Role: LEADER
I20260925 00:54:13.151314 27468 consensus_queue.cc:237] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88f3b20efdb4184ba214906a579253d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 35989 } }
I20260925 00:54:13.152055 27463 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:54:13.153240 27469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a88f3b20efdb4184ba214906a579253d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88f3b20efdb4184ba214906a579253d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 35989 } } }
I20260925 00:54:13.153379 27469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:13.153681 27470 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a88f3b20efdb4184ba214906a579253d. Latest consensus state: current_term: 1 leader_uuid: "a88f3b20efdb4184ba214906a579253d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88f3b20efdb4184ba214906a579253d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 35989 } } }
I20260925 00:54:13.153774 27470 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:13.154968 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.140102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.16.202.62:35989)
0925 00:54:13.140593 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.140611 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.140652 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:54:13.141494 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.141515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.141903 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.142278 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:13.143164 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.144225 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.145158 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.145672 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.145835 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.148490 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:13.148497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:13.148516 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:13.148553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:13.150520 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.151089 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.151096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.151102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.151251 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.151664 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.151667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.151669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.152228 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:13.152449 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.153230 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.153912 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":132,"threads_started":1}
W20260925 00:54:13.157388 27483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:54:13.157464 27483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:54:13.157518 27477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:54:13.158203 27477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:54:13.164044 27477 catalog_manager.cc:1347] Generated new cluster ID: 3249f2c26c0a4b45914af2f924f5bcf9
I20260925 00:54:13.164131 27477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:54:13.176697 27477 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:54:13.178048 27477 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:54:13.188740 27477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: Generated new TSK 0
I20260925 00:54:13.189476 27477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:54:13.227658 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:35989
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:54:13.339033 27489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:13.339306 27489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:13.339368 27489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:13.343407 27489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:13.343468 27489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:13.343552 27489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:54:13.348966 27489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:13.350055 27489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:13.351089 27489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:54:13.360747 27489 server_base.cc:1034] running on GCE node
W20260925 00:54:13.360714 27495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.360706 27497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.360706 27494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:13.361318 27489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:13.361979 27489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:13.363121 27489 hybrid_clock.cc:648] HybridClock initialized: now 1790297653363102 us; error 32 us; skew 500 ppm
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:13.365881 27489 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:54:13.366751 27489 webserver.cc:460] Webserver started at http://127.16.202.1:44571/ using document root <none> and password file <none>
I20260925 00:54:13.367290 27489 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:13.367363 27489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:13.367565 27489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:13.369649 27489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "18a691579fd94a5db1eb91facfc520ad"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "659e1b97c221a7de70761a7fc34d6b80"
server_key_iv: "92c1393f5f43e44f15cbc0c72293f69a"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.370203 27489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "18a691579fd94a5db1eb91facfc520ad"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "659e1b97c221a7de70761a7fc34d6b80"
server_key_iv: "92c1393f5f43e44f15cbc0c72293f69a"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.374109 27489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:54:13.376499 27504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.377882 27489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:54:13.378013 27489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "18a691579fd94a5db1eb91facfc520ad"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "659e1b97c221a7de70761a7fc34d6b80"
server_key_iv: "92c1393f5f43e44f15cbc0c72293f69a"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.378129 27489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:13.389509 27489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:13.392565 27489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:13.392758 27489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:13.393345 27489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:13.394148 27489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:13.394193 27489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.394244 27489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:13.394273 27489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.402640 27489 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:46297
I20260925 00:54:13.402676 27617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:46297 every 8 connection(s)
I20260925 00:54:13.403563 27489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:54:13.403939 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 27489
I20260925 00:54:13.404048 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:54:13.404346 17192 external_mini_cluster.cc:1428] Setting key 4fb431bde80b8df45a5c3055e96741aa
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:54:13.419437 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.405490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:47585 (local address 127.16.202.62:35989)
0925 00:54:13.405628 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.405632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.406719 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 00:54:13.408050 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.408053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.408106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.408178 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:13.409374 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.409899 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.411002 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.411229 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.411317 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.415999 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:13.416003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:13.416006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:13.416036 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:13.417481 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.417966 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.417970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.417973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.418053 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.418359 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.418363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.418364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.418550 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:13.418639 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.419142 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.419269 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:54:13.420269 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.406023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:35989 (local address 127.16.202.1:47585)
0925 00:54:13.406564 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:13.406611 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:13.407833 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:13.408355 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:13.408363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:13.408745 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:13.409231 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.409244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.410014 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.410017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.410876 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.410884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.411364 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.411368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.411528 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.412060 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:13.412076 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:13.415814 (+ 3738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:13.417591 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:13.417596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:13.417610 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:13.417866 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:13.418133 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:13.418136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:13.418138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:13.418273 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:13.418648 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:13.418654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:13.418913 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 00:54:13.419085 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 00:54:13.419433 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":149,"threads_started":1}
I20260925 00:54:13.421604 27618 heartbeater.cc:347] Connected to a master server at 127.16.202.62:35989
I20260925 00:54:13.421832 27618 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:13.422333 27618 heartbeater.cc:511] Master 127.16.202.62:35989 requested a full tablet report, sending...
I20260925 00:54:13.423988 27427 ts_manager.cc:194] Registered new tserver with Master: 18a691579fd94a5db1eb91facfc520ad (127.16.202.1:46297)
I20260925 00:54:13.425256 27427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:47585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:54:13.431483 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:35989
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:54:13.543128 27623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:13.543448 27623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:13.543499 27623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:13.547685 27623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:13.547761 27623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:13.547842 27623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:54:13.553375 27623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:13.554508 27623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:13.555550 27623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:13.564640 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.564730 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.564641 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:13.564841 27623 server_base.cc:1034] running on GCE node
I20260925 00:54:13.565343 27623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:13.565963 27623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:13.567126 27623 hybrid_clock.cc:648] HybridClock initialized: now 1790297653567113 us; error 34 us; skew 500 ppm
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:13.569934 27623 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:54:13.570932 27623 webserver.cc:460] Webserver started at http://127.16.202.2:40561/ using document root <none> and password file <none>
I20260925 00:54:13.571520 27623 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:13.571600 27623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:13.571830 27623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:13.573995 27623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "0dfd69c4e08d40f196cf6667aad0f29a"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "5663b5549813c30c614a761b10d68fee"
server_key_iv: "107705c11cee23cf10fd0e53b58ce5a6"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.574582 27623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "0dfd69c4e08d40f196cf6667aad0f29a"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "5663b5549813c30c614a761b10d68fee"
server_key_iv: "107705c11cee23cf10fd0e53b58ce5a6"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.578644 27623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 00:54:13.581426 27638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.582969 27623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:54:13.583107 27623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "0dfd69c4e08d40f196cf6667aad0f29a"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "5663b5549813c30c614a761b10d68fee"
server_key_iv: "107705c11cee23cf10fd0e53b58ce5a6"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.583222 27623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:13.610028 27623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:13.613197 27623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:13.613377 27623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:13.613961 27623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:13.614769 27623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:13.614815 27623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.614866 27623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:13.614895 27623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.623260 27623 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:41883
I20260925 00:54:13.623313 27751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:41883 every 8 connection(s)
I20260925 00:54:13.624166 27623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:54:13.628374 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 27623
I20260925 00:54:13.628485 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:54:13.628786 17192 external_mini_cluster.cc:1428] Setting key 7c499f7eb239e9264b605c313afca5c4
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:54:13.637710 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.625984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:51683 (local address 127.16.202.62:35989)
0925 00:54:13.626126 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.626130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.626977 (+ 847us) server_negotiation.cc:407] Connection header received
0925 00:54:13.628318 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.628321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.628378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.628446 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:13.629740 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.630327 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.631446 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.631654 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.631740 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.634115 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:13.634118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:13.634120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:13.634146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:13.635626 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.636160 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.636163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.636164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.636241 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.636566 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.636569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.636571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.636802 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:13.636891 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.637414 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.637547 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:54:13.638583 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.626349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:35989 (local address 127.16.202.2:51683)
0925 00:54:13.626822 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:13.626866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:13.628117 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:13.628613 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:13.628620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:13.629039 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:13.629605 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.629619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.630445 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.630448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.631334 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.631343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.631788 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.631792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.631963 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.632493 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:13.632508 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:13.633945 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:13.635751 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:13.635757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:13.635774 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:13.636054 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:13.636324 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:13.636327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:13.636329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:13.636480 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:13.636898 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:13.636905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:13.637177 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 00:54:13.637351 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 00:54:13.637693 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:54:13.639745 27752 heartbeater.cc:347] Connected to a master server at 127.16.202.62:35989
I20260925 00:54:13.639989 27752 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:13.640494 27752 heartbeater.cc:511] Master 127.16.202.62:35989 requested a full tablet report, sending...
I20260925 00:54:13.641621 27427 ts_manager.cc:194] Registered new tserver with Master: 0dfd69c4e08d40f196cf6667aad0f29a (127.16.202.2:41883)
I20260925 00:54:13.642416 27427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:51683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:54:13.655586 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:35989
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:54:13.768520 27757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:13.768853 27757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:13.768908 27757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:13.773440 27757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:13.773517 27757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:13.773591 27757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:54:13.779148 27757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:13.780247 27757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:13.781363 27757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:13.791438 27765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.791487 27763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:13.791369 27762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:13.791834 27757 server_base.cc:1034] running on GCE node
I20260925 00:54:13.792223 27757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:13.792927 27757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:13.794093 27757 hybrid_clock.cc:648] HybridClock initialized: now 1790297653794060 us; error 42 us; skew 500 ppm
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:13.796811 27757 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:54:13.797636 27757 webserver.cc:460] Webserver started at http://127.16.202.3:44409/ using document root <none> and password file <none>
I20260925 00:54:13.798177 27757 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:13.798237 27757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:13.798445 27757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:54:13.800530 27757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "ed6730a5dad541e5924748b5174c3eec"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "d237d5e60b470a78232a8550c5c8e261"
server_key_iv: "89ab32d61b939c89f14625b1fd9f156e"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.801136 27757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "ed6730a5dad541e5924748b5174c3eec"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "d237d5e60b470a78232a8550c5c8e261"
server_key_iv: "89ab32d61b939c89f14625b1fd9f156e"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.805182 27757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:54:13.807700 27772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.808969 27757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:54:13.809072 27757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "ed6730a5dad541e5924748b5174c3eec"
format_stamp: "Formatted at 2026-09-25 00:54:13 on dist-test-slave-rpk1"
server_key: "d237d5e60b470a78232a8550c5c8e261"
server_key_iv: "89ab32d61b939c89f14625b1fd9f156e"
server_key_version: "encryptionkey@0"
I20260925 00:54:13.809176 27757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:13.829341 27757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:13.832142 27757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:13.832316 27757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:13.832937 27757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:13.833775 27757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:13.833820 27757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.833864 27757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:13.833894 27757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:13.842093 27757 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:43491
I20260925 00:54:13.842126 27885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:43491 every 8 connection(s)
I20260925 00:54:13.843003 27757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
Sep 25 00:54:13 dist-test-slave-rpk1 krb5kdc[27382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297653, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:54:13.852507 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 27757
I20260925 00:54:13.852613 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:54:13.852952 17192 external_mini_cluster.cc:1428] Setting key f81dffcc216d20520900af7aefe2c84b
I20260925 00:54:13.856626 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.844840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:57559 (local address 127.16.202.62:35989)
0925 00:54:13.844994 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.844999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.845925 (+ 926us) server_negotiation.cc:407] Connection header received
0925 00:54:13.847257 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.847260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.847314 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.847401 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:13.848572 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.849174 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.850264 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.850492 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.850576 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.853137 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:13.853141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:13.853144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:13.853174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:13.854721 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.855262 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.855265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.855266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.855350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.855638 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.855640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.855641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.855825 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:13.855899 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.856378 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.856490 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:54:13.857492 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.845306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:35989 (local address 127.16.202.3:57559)
0925 00:54:13.845775 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:13.845819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:13.847053 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:13.847574 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:13.847582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:13.847945 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:13.848433 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.848446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.849286 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.849289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.850155 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.850164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.850611 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.850613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.850754 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.851301 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:13.851317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:13.852974 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:13.854827 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:13.854831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:13.854849 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:13.855171 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:13.855421 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:13.855423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:13.855425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:13.855559 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:13.855898 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:13.855903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:13.856139 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 00:54:13.856295 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 00:54:13.856594 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260925 00:54:13.858525 27886 heartbeater.cc:347] Connected to a master server at 127.16.202.62:35989
I20260925 00:54:13.858742 27886 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:13.859200 27886 heartbeater.cc:511] Master 127.16.202.62:35989 requested a full tablet report, sending...
I20260925 00:54:13.860181 27427 ts_manager.cc:194] Registered new tserver with Master: ed6730a5dad541e5924748b5174c3eec (127.16.202.3:43491)
I20260925 00:54:13.860770 27427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:57559
I20260925 00:54:13.866926 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:54:13.875358 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.868761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.16.202.62:35989)
0925 00:54:13.868879 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.868883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.869021 (+ 138us) server_negotiation.cc:407] Connection header received
0925 00:54:13.869104 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.869106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.869147 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.869208 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:13.869819 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.870337 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.871109 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.871360 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.871470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.872692 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:13.872697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:13.872700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:13.872723 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:13.873872 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.874224 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.874226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.874227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.874298 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:13.874507 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:13.874509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:13.874511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:13.874757 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:13.874860 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.875145 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.875237 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:54:13.878180 27427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:54:13.881372 27427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:54:13.896008 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.890296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:41883 (local address 127.0.0.1:53476)
0925 00:54:13.890695 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:13.890758 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:13.890886 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:13.892268 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:13.892271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:13.892306 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:13.892459 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.892466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.893486 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.893489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.894608 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.894618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.895436 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.895439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.895556 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.895594 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 00:54:13.895708 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 00:54:13.895845 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":18,"spinlock_wait_cycles":5248,"thread_start_us":86,"threads_started":1}
I20260925 00:54:13.896011 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.890469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.16.202.2:41883)
0925 00:54:13.891771 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.891784 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.891804 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:54:13.891883 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.891898 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.892106 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.892374 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:13.892531 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.893362 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.894739 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.895315 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.895424 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.895608 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.895754 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.895853 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":133,"threads_started":1}
I20260925 00:54:13.896936 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.891060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46297 (local address 127.0.0.1:39598)
0925 00:54:13.891629 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:13.891641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:13.891721 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:13.892432 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:13.892435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:13.892449 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:13.892545 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.892551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.894017 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.894020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.895396 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.895407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.896491 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.896495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.896586 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.896610 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:54:13.896707 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 00:54:13.896781 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":137,"thread_start_us":86,"threads_started":1}
I20260925 00:54:13.896989 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.891249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.16.202.1:46297)
0925 00:54:13.891582 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.891600 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.891808 (+ 208us) server_negotiation.cc:407] Connection header received
0925 00:54:13.891883 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.891889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.892106 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.892259 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:13.892661 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.893867 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.895533 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.896380 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.896476 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.896616 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.896747 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.896842 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":128,"threads_started":1}
I20260925 00:54:13.897415 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.890574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43491 (local address 127.0.0.1:39488)
0925 00:54:13.891411 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:13.891425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:13.891510 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:13.892261 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:13.892267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:13.892303 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:13.892587 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.892590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.894020 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.894022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.895400 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:13.895407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.896908 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.896912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:13.897124 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.897148 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:54:13.897227 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 00:54:13.897294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":61,"spinlock_wait_cycles":7680,"thread_start_us":99,"threads_started":1}
I20260925 00:54:13.898142 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:13.891029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.16.202.3:43491)
0925 00:54:13.891783 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:13.891795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:13.891816 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:54:13.891892 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:13.891898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:13.892106 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:13.892258 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:13.892711 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.893867 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.895533 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:13.896771 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:13.897720 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:13.897866 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:13.897914 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 00:54:13.897999 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":116,"threads_started":1}
I20260925 00:54:13.899395 27686 tablet_service.cc:1467] Processing CreateTablet for tablet b0ac61cef68546f48ca60d55b187a49b (DEFAULT_TABLE table=test-table [id=06f7cf5f500542e6bd16524225fd0127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:54:13.900388 27686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ac61cef68546f48ca60d55b187a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:13.903256 27820 tablet_service.cc:1467] Processing CreateTablet for tablet b0ac61cef68546f48ca60d55b187a49b (DEFAULT_TABLE table=test-table [id=06f7cf5f500542e6bd16524225fd0127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:54:13.903254 27552 tablet_service.cc:1467] Processing CreateTablet for tablet b0ac61cef68546f48ca60d55b187a49b (DEFAULT_TABLE table=test-table [id=06f7cf5f500542e6bd16524225fd0127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:54:13.904160 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ac61cef68546f48ca60d55b187a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:13.904170 27820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ac61cef68546f48ca60d55b187a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:13.910081 27903 tablet_bootstrap.cc:492] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Bootstrap starting.
I20260925 00:54:13.911455 27904 tablet_bootstrap.cc:492] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: Bootstrap starting.
I20260925 00:54:13.912341 27905 tablet_bootstrap.cc:492] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Bootstrap starting.
I20260925 00:54:13.912420 27903 tablet_bootstrap.cc:654] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:13.913727 27903 log.cc:824] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:13.913807 27904 tablet_bootstrap.cc:654] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:13.914845 27905 tablet_bootstrap.cc:654] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:13.915021 27904 log.cc:824] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:13.915689 27905 log.cc:824] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:13.915861 27903 tablet_bootstrap.cc:492] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: No bootstrap required, opened a new log
I20260925 00:54:13.916064 27903 ts_tablet_manager.cc:1397] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:54:13.917019 27904 tablet_bootstrap.cc:492] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: No bootstrap required, opened a new log
I20260925 00:54:13.917196 27904 ts_tablet_manager.cc:1397] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 00:54:13.917295 27905 tablet_bootstrap.cc:492] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: No bootstrap required, opened a new log
I20260925 00:54:13.917469 27905 ts_tablet_manager.cc:1397] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:54:13.919929 27903 raft_consensus.cc:348] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:13.920058 27905 raft_consensus.cc:348] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:13.920212 27903 raft_consensus.cc:374] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:13.920253 27905 raft_consensus.cc:374] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:13.920276 27903 raft_consensus.cc:729] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6730a5dad541e5924748b5174c3eec, State: Initialized, Role: FOLLOWER
I20260925 00:54:13.920296 27905 raft_consensus.cc:729] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfd69c4e08d40f196cf6667aad0f29a, State: Initialized, Role: FOLLOWER
I20260925 00:54:13.920740 27905 consensus_queue.cc:260] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:13.920739 27903 consensus_queue.cc:260] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:13.920785 27904 raft_consensus.cc:348] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:13.921006 27904 raft_consensus.cc:374] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:13.921077 27904 raft_consensus.cc:729] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a691579fd94a5db1eb91facfc520ad, State: Initialized, Role: FOLLOWER
I20260925 00:54:13.921506 27904 consensus_queue.cc:260] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:13.921754 27903 ts_tablet_manager.cc:1428] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:54:13.921798 27905 ts_tablet_manager.cc:1428] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 00:54:13.922094 27886 heartbeater.cc:503] Master 127.16.202.62:35989 was elected leader, sending a full tablet report...
I20260925 00:54:13.922101 27752 heartbeater.cc:503] Master 127.16.202.62:35989 was elected leader, sending a full tablet report...
I20260925 00:54:13.922290 27904 ts_tablet_manager.cc:1428] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:54:13.922603 27618 heartbeater.cc:503] Master 127.16.202.62:35989 was elected leader, sending a full tablet report...
W20260925 00:54:14.093986 27887 tablet.cc:2369] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:54:14.095527 27910 raft_consensus.cc:482] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:54:14.095734 27910 raft_consensus.cc:504] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:14.096923 27910 leader_election.cc:288] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a691579fd94a5db1eb91facfc520ad (127.16.202.1:46297), ed6730a5dad541e5924748b5174c3eec (127.16.202.3:43491)
I20260925 00:54:14.100353 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:14.097147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46297 (local address 127.16.202.2:44161)
0925 00:54:14.097269 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:14.097283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:14.097385 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:14.097660 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:14.097662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:14.097706 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:14.097821 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:14.097827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.098524 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.098527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:14.099346 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:14.099355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.099947 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.099950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:14.100053 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:14.100075 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:54:14.100141 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 00:54:14.100213 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":18,"spinlock_wait_cycles":9728}
I20260925 00:54:14.100353 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:14.097201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:44161 (local address 127.16.202.1:46297)
0925 00:54:14.097325 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:14.097330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:14.097358 (+ 28us) server_negotiation.cc:407] Connection header received
0925 00:54:14.097523 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:14.097527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:14.097573 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:14.097635 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:14.097938 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.098428 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.099467 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.099843 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.099947 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:14.099999 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:14.100170 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 00:54:14.100236 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:54:14.100549 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:14.097418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:57441 (local address 127.16.202.3:43491)
0925 00:54:14.097525 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:14.097529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:14.097663 (+ 134us) server_negotiation.cc:407] Connection header received
0925 00:54:14.097801 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:14.097804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:14.097869 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:14.097930 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:14.098178 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.098673 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.099662 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.100060 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.100162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:14.100209 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:14.100368 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 00:54:14.100437 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:54:14.100569 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:14.097307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43491 (local address 127.16.202.2:57441)
0925 00:54:14.097578 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:14.097588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:14.097710 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:14.097954 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:14.097957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:14.097971 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:14.098082 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:14.098087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.098777 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.098780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:14.099554 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:14.099562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.100165 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.100169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:14.100267 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:14.100284 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:54:14.100353 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:54:14.100425 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3,"spinlock_wait_cycles":20096,"thread_start_us":91,"threads_started":1}
I20260925 00:54:14.101259 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ac61cef68546f48ca60d55b187a49b" candidate_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a691579fd94a5db1eb91facfc520ad" is_pre_election: true
I20260925 00:54:14.101275 27840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ac61cef68546f48ca60d55b187a49b" candidate_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6730a5dad541e5924748b5174c3eec" is_pre_election: true
I20260925 00:54:14.101542 27840 raft_consensus.cc:2454] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfd69c4e08d40f196cf6667aad0f29a in term 0.
I20260925 00:54:14.101547 27572 raft_consensus.cc:2454] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfd69c4e08d40f196cf6667aad0f29a in term 0.
I20260925 00:54:14.101971 27639 leader_election.cc:302] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfd69c4e08d40f196cf6667aad0f29a, 18a691579fd94a5db1eb91facfc520ad; no voters:
I20260925 00:54:14.102222 27910 raft_consensus.cc:2781] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:54:14.102360 27910 raft_consensus.cc:482] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:54:14.102407 27910 raft_consensus.cc:3037] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:14.103428 27910 raft_consensus.cc:504] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:14.103760 27910 leader_election.cc:288] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [CANDIDATE]: Term 1 election: Requested vote from peers 18a691579fd94a5db1eb91facfc520ad (127.16.202.1:46297), ed6730a5dad541e5924748b5174c3eec (127.16.202.3:43491)
I20260925 00:54:14.104135 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ac61cef68546f48ca60d55b187a49b" candidate_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a691579fd94a5db1eb91facfc520ad"
I20260925 00:54:14.104135 27840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ac61cef68546f48ca60d55b187a49b" candidate_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6730a5dad541e5924748b5174c3eec"
I20260925 00:54:14.104255 27840 raft_consensus.cc:3037] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:14.104255 27572 raft_consensus.cc:3037] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:14.105546 27572 raft_consensus.cc:2454] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfd69c4e08d40f196cf6667aad0f29a in term 1.
I20260925 00:54:14.105546 27840 raft_consensus.cc:2454] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfd69c4e08d40f196cf6667aad0f29a in term 1.
I20260925 00:54:14.105804 27639 leader_election.cc:302] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfd69c4e08d40f196cf6667aad0f29a, 18a691579fd94a5db1eb91facfc520ad; no voters:
I20260925 00:54:14.105975 27910 raft_consensus.cc:2781] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:54:14.106189 27910 raft_consensus.cc:686] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 1 LEADER]: Becoming Leader. State: Replica: 0dfd69c4e08d40f196cf6667aad0f29a, State: Running, Role: LEADER
I20260925 00:54:14.106546 27910 consensus_queue.cc:237] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } }
I20260925 00:54:14.109674 27427 catalog_manager.cc:5495] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfd69c4e08d40f196cf6667aad0f29a (127.16.202.2). New cstate: current_term: 1 leader_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfd69c4e08d40f196cf6667aad0f29a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 41883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 } health_report { overall_health: UNKNOWN } } }
W20260925 00:54:14.125324 27753 tablet.cc:2369] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:54:14.134743 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:14.130699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.16.202.2:41883)
0925 00:54:14.130851 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:14.130856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:14.130872 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:54:14.130952 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:14.130955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:14.131021 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:14.131117 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:54:14.131311 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.132098 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.133024 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:14.133377 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:14.133493 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:14.133721 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:54:14.134227 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:54:14.134347 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:14.134462 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 00:54:14.134525 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:54:14.141455 27840 raft_consensus.cc:1261] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 1 FOLLOWER]: Refusing update from remote peer 0dfd69c4e08d40f196cf6667aad0f29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:54:14.141444 27572 raft_consensus.cc:1261] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad [term 1 FOLLOWER]: Refusing update from remote peer 0dfd69c4e08d40f196cf6667aad0f29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:54:14.142266 27910 consensus_queue.cc:1035] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a691579fd94a5db1eb91facfc520ad" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:54:14.142493 27913 consensus_queue.cc:1035] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6730a5dad541e5924748b5174c3eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:54:14.150945 27917 mvcc.cc:205] Tried to move back new op lower bound from 7333059191354974208 to 7333059191222951936. Current Snapshot: MvccSnapshot[applied={T|T < 7333059191354974208}]
I20260925 00:54:14.160174 27427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:54:14.160348 27427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:54:14.162999 27427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a88f3b20efdb4184ba214906a579253d: Sending DeleteTablet for 3 replicas of tablet b0ac61cef68546f48ca60d55b187a49b
I20260925 00:54:14.163854 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ac61cef68546f48ca60d55b187a49b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:54:14 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260925 00:54:14.163856 27820 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ac61cef68546f48ca60d55b187a49b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:54:14 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:39488
I20260925 00:54:14.164423 27927 tablet_replica.cc:330] T b0ac61cef68546f48ca60d55b187a49b P 18a691579fd94a5db1eb91facfc520ad: stopping tablet replica
I20260925 00:54:14.164413 27686 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ac61cef68546f48ca60d55b187a49b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:54:14 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:53476
I20260925 00:54:14.164758 27928 tablet_replica.cc:330] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: stopping tablet replica
I20260925 00:54:14.165042 27929 tablet_replica.cc:330] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: stopping tablet replica
I20260925 00:54:14.165365 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 27489
I20260925 00:54:14.165361 27929 raft_consensus.cc:2229] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:54:14.165964 27928 raft_consensus.cc:2229] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:54:14.166435 27928 raft_consensus.cc:2258] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:54:14.166798 27929 raft_consensus.cc:2258] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:54:14.168514 27928 ts_tablet_manager.cc:1905] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:54:14.168519 27929 ts_tablet_manager.cc:1905] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:54:14.171375 27929 ts_tablet_manager.cc:1918] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:54:14.171479 27929 log.cc:1196] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/b0ac61cef68546f48ca60d55b187a49b
I20260925 00:54:14.171422 27928 ts_tablet_manager.cc:1918] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:54:14.171697 27928 log.cc:1196] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/b0ac61cef68546f48ca60d55b187a49b
I20260925 00:54:14.171761 27929 ts_tablet_manager.cc:1939] T b0ac61cef68546f48ca60d55b187a49b P 0dfd69c4e08d40f196cf6667aad0f29a: Deleting consensus metadata
I20260925 00:54:14.172336 27928 ts_tablet_manager.cc:1939] T b0ac61cef68546f48ca60d55b187a49b P ed6730a5dad541e5924748b5174c3eec: Deleting consensus metadata
I20260925 00:54:14.173216 27411 catalog_manager.cc:4841] TS 0dfd69c4e08d40f196cf6667aad0f29a (127.16.202.2:41883): tablet b0ac61cef68546f48ca60d55b187a49b (table test-table [id=06f7cf5f500542e6bd16524225fd0127]) successfully deleted
I20260925 00:54:14.173625 27414 catalog_manager.cc:4841] TS ed6730a5dad541e5924748b5174c3eec (127.16.202.3:43491): tablet b0ac61cef68546f48ca60d55b187a49b (table test-table [id=06f7cf5f500542e6bd16524225fd0127]) successfully deleted
W20260925 00:54:14.174028 27411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:46297 (error 108)
I20260925 00:54:14.174232 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 27623
I20260925 00:54:14.175516 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:14.175182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46297 (local address 127.0.0.1:39614)
0925 00:54:14.175361 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:14.175429 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 00:54:14.175777 27411 catalog_manager.cc:4584] TS 18a691579fd94a5db1eb91facfc520ad (127.16.202.1:46297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0ac61cef68546f48ca60d55b187a49b: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46297: connect: Connection refused (error 111)
I20260925 00:54:14.179494 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 27757
I20260925 00:54:14.184162 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 27395
2026-09-25T00:54:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2313 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:54:14 dist-test-slave-rpk1 krb5kdc[27933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:54:14 dist-test-slave-rpk1 krb5kdc[27933](info): set up 2 sockets
Sep 25 00:54:14 dist-test-slave-rpk1 krb5kdc[27933](info): commencing operation
krb5kdc: starting...
W20260925 00:54:15.141472 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:54:15 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:54:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/rangeradmin_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/rangeradmin_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/rangerlookup_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/rangerlookup_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
I20260925 00:54:15.323649 17192 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/postgres -l logfile start
2026-09-25 00:54:17.867 UTC [27963] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 00:54:17.867 UTC [27963] LOG: listening on IPv4 address "127.16.202.20", port 37029
2026-09-25 00:54:17.870 UTC [27963] LOG: listening on Unix socket "/tmp/.s.PGSQL.37029"
2026-09-25 00:54:17.900 UTC [27966] LOG: database system was shut down at 2026-09-25 00:54:16 UTC
2026-09-25 00:54:17.904 UTC [27963] LOG: database system is ready to accept connections
I20260925 00:54:18.761543 17192 mini_postgres.cc:96] Postgres bound to 37029
2026-09-25 00:54:18.767 UTC [27974] FATAL: database "slave" does not exist
127.16.202.20:37029 - accepting connections
I20260925 00:54:18.768299 17192 mini_ranger.cc:161] Starting Ranger...
I20260925 00:54:18.782894 17192 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 00:54:19.269344 17192 mini_ranger.cc:87] Created ranger Postgres database
I20260925 00:54:19.269464 17192 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-admin
2026-09-25 00:54:19,538 [I] DB FLAVOR :POSTGRES
2026-09-25 00:54:19,539 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:54:19,539 [I] Checking connection..
2026-09-25 00:54:19,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 00:54:19,785 [I] Checking connection passed.
2026-09-25 00:54:19,786 [I] --------- Verifying version history table ---------
2026-09-25 00:54:19,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:54:20,031 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:54:20,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:54:20,290 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:54:20,290 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 00:54:20,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 00:54:20.530 UTC [28038] WARNING: there is no transaction in progress
2026-09-25 00:54:20,536 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 00:54:20,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:54:20,784 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 00:54:20,784 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:54:20,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 00:54:21,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 00:54:21,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rpk1.c.gcp-upstream.internal','N') ;"
2026-09-25 00:54:21,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:54:21,773 [I] Table x_portal_user does not exist in database ranger
2026-09-25 00:54:21,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:54:22,020 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 00:54:22,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:54:22,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:54:22,503 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 00:54:22,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 00:54:23.476 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.493 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.510 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.526 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.543 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.562 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.624 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.631 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.647 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.657 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.667 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.681 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.691 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.697 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.703 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:23.710 UTC [28146] WARNING: there is no transaction in progress
2026-09-25 00:54:24,369 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 00:54:24,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 00:54:24,630 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 00:54:24,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:54:24,868 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 00:54:24,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:54:25,111 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 00:54:25,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:54:25,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:54:25,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rpk1.c.gcp-upstream.internal';"
2026-09-25 00:54:25,870 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 00:54:25,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 00:54:26,115 [I] DB_PATCHES have already been applied
I20260925 00:54:26.120523 17192 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 00:54:26.120589 17192 mini_ranger.cc:193] Using host: 127.16.202.20
I20260925 00:54:26.123955 17192 mini_ranger.cc:227] Ranger admin URL: http://127.16.202.20:50855
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 00:54:27.124789 17192 mini_ranger.cc:160] Time spent starting Ranger: real 8.357s user 0.000s sys 0.009s
Sep 25, 2026 12:54:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:54:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 12:54:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:54:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:54:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 00:54:49.580868 17192 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/rangerkms_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/rangerkms_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.202.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.202.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 00:54:49 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297689, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 00:54:49.669200 17192 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 00:54:49.679994 17192 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 00:54:50.641343 17192 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 00:54:50.641500 17192 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms
I20260925 00:54:50.641600 17192 mini_ranger_kms.cc:227] Using postgres at 127.16.202.20:37029
2026-09-25 00:54:50,819 [I] DB FLAVOR :POSTGRES
2026-09-25 00:54:50,819 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:54:50,819 [I] Checking connection
2026-09-25 00:54:50,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 00:54:51,066 [I] connection success
2026-09-25 00:54:51,067 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 00:54:51,067 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 00:54:51,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 00:54:51,311 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 00:54:51,312 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:54:51,312 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 00:54:51,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:37029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 00:54:51,567 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 00:54:52.090778 17192 mini_ranger_kms.cc:310] Created kms service
I20260925 00:54:52.316438 17192 mini_ranger_kms.cc:326] Created kudu user
I20260925 00:54:52.365090 17192 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 00:54:52 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297689, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:54:52.751032 17192 mini_ranger_kms.cc:384] Added ranger policy
I20260925 00:54:52.751190 17192 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 00:54:52.751209 17192 mini_ranger_kms.cc:240] Using host: 127.16.202.20
I20260925 00:54:52.755023 17192 mini_ranger_kms.cc:276] Ranger KMS PID: 28326
I20260925 00:54:52.755165 17192 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.202.20:55535
Sep 25, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 00:54:54.785012 17192 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.116s user 0.020s sys 0.143s
I20260925 00:54:54.785140 17192 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 00:54:54.785251 17192 mini_ranger_kms.cc:401] 127.16.202.20:55535/kms/v1/keys
Sep 25 00:54:54 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.202.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 00:54:55 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.202.20@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
Sep 25 00:54:55 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.202.20@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
Sep 25 00:54:57 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:54:57.917794 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33027
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:33027
--ranger_config_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:55535/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:54:58.032038 28377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:58.032331 28377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:58.032370 28377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:58.036548 28377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:54:58.036623 28377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:58.036639 28377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:58.036654 28377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:54:58.036695 28377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:54:58.042363 28377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.202.20:55535/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:33027
--ranger_config_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:58.043502 28377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:58.044574 28377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:58.053890 28383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:58.053884 28382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:58.054056 28377 server_base.cc:1034] running on GCE node
W20260925 00:54:58.054241 28385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:58.054804 28377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:58.056017 28377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:58.057184 28377 hybrid_clock.cc:648] HybridClock initialized: now 1790297698057165 us; error 34 us; skew 500 ppm
Sep 25 00:54:58 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297698, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:58.059911 28377 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:54:58.060956 28377 webserver.cc:460] Webserver started at http://127.16.202.62:39919/ using document root <none> and password file <none>
I20260925 00:54:58.061553 28377 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:58.061617 28377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:58.061817 28377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:54:58 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297698, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:54:58.127039 28377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "622d1db84611452ab50c788780953ed8"
format_stamp: "Formatted at 2026-09-25 00:54:58 on dist-test-slave-rpk1"
server_key: "1fe0f4b487160a65ebb706829643241b"
server_key_iv: "e2fd54ec1d0673877bb86b1014b531b3"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:58.127792 28377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "622d1db84611452ab50c788780953ed8"
format_stamp: "Formatted at 2026-09-25 00:54:58 on dist-test-slave-rpk1"
server_key: "1fe0f4b487160a65ebb706829643241b"
server_key_iv: "e2fd54ec1d0673877bb86b1014b531b3"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:58.133015 28377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 00:54:58.151041 28392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:58.152943 28377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:54:58.153123 28377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "622d1db84611452ab50c788780953ed8"
format_stamp: "Formatted at 2026-09-25 00:54:58 on dist-test-slave-rpk1"
server_key: "1fe0f4b487160a65ebb706829643241b"
server_key_iv: "e2fd54ec1d0673877bb86b1014b531b3"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:58.153273 28377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:58.172060 28377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:58.175457 28377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:58.175660 28377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:58.184230 28377 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:33027
I20260925 00:54:58.184274 28444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:33027 every 8 connection(s)
I20260925 00:54:58.185619 28377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:54:58.188795 28445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:58.194579 28445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8: Bootstrap starting.
I20260925 00:54:58.195814 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 28377
I20260925 00:54:58.195937 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:54:58.196244 17192 external_mini_cluster.cc:1428] Setting key 35cade9ead3c204fc19d2ca8bc690e31
I20260925 00:54:58.197535 28445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:58.198361 28445 log.cc:824] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:58.200219 28445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8: No bootstrap required, opened a new log
I20260925 00:54:58.204303 28445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622d1db84611452ab50c788780953ed8" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33027 } }
I20260925 00:54:58.204571 28445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:58.204648 28445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d1db84611452ab50c788780953ed8, State: Initialized, Role: FOLLOWER
I20260925 00:54:58.205181 28445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622d1db84611452ab50c788780953ed8" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33027 } }
I20260925 00:54:58.205346 28445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:54:58.205443 28445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:54:58.205546 28445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:58.206749 28445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622d1db84611452ab50c788780953ed8" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33027 } }
I20260925 00:54:58.207129 28445 leader_election.cc:302] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 622d1db84611452ab50c788780953ed8; no voters:
Sep 25 00:54:58 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:54:58.207496 28445 leader_election.cc:288] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:54:58.207535 28450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:54:58.207722 28450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 622d1db84611452ab50c788780953ed8, State: Running, Role: LEADER
I20260925 00:54:58.208017 28450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622d1db84611452ab50c788780953ed8" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33027 } }
I20260925 00:54:58.208851 28445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:54:58.209286 28450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 622d1db84611452ab50c788780953ed8. Latest consensus state: current_term: 1 leader_uuid: "622d1db84611452ab50c788780953ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622d1db84611452ab50c788780953ed8" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33027 } } }
I20260925 00:54:58.209429 28450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:58.209640 28450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "622d1db84611452ab50c788780953ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622d1db84611452ab50c788780953ed8" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33027 } } }
I20260925 00:54:58.209754 28450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8 [sys.catalog]: This master's current role is: LEADER
I20260925 00:54:58.210966 28445 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 00:54:58.212217 28454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:54:58.218495 28454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:54:58.219390 28448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:58.197468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.16.202.62:33027)
0925 00:54:58.198711 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:58.198730 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:58.198775 (+ 45us) server_negotiation.cc:407] Connection header received
0925 00:54:58.199652 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:58.199677 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:58.200131 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:58.200556 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:58.201415 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:58.202870 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:58.204079 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:58.204883 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:58.205045 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:58.208280 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:58.208288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:58.208308 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:58.208342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:58.210772 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:58.211347 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:58.211353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:58.211360 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:58.211492 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:58.211805 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:58.211808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:58.211810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:58.212371 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:58.214582 (+ 2211us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:58.214956 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 00:54:58.215832 (+ 876us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":145,"threads_started":1}
I20260925 00:54:58.228989 28454 catalog_manager.cc:1347] Generated new cluster ID: 1fbc5cb8963e4c10aa2bee2cf321af55
I20260925 00:54:58.229153 28454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:54:58.241689 28454 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:54:58.243156 28454 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:54:58.260555 28454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8: Generated new TSK 0
I20260925 00:54:58.261628 28454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:54:58.900120 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33027
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:54:59.042608 28490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:59.043028 28490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:59.043083 28490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:59.047520 28490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:59.047596 28490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:59.047680 28490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:54:59.053578 28490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.202.20:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:59.054746 28490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:59.055816 28490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:59.066946 28498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:59.067390 28490 server_base.cc:1034] running on GCE node
W20260925 00:54:59.066916 28500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:59.067731 28497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:59.068288 28490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:59.069190 28490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:59.070422 28490 hybrid_clock.cc:648] HybridClock initialized: now 1790297699070390 us; error 49 us; skew 500 ppm
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:59.074376 28490 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:54:59.075575 28490 webserver.cc:460] Webserver started at http://127.16.202.1:34003/ using document root <none> and password file <none>
I20260925 00:54:59.076306 28490 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:59.076399 28490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:59.076728 28490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:54:59.093449 28490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "51d99dcff77f403b8fa9a08db02ca0b4"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "8892820ebf29d53910a8594c3952db9f"
server_key_iv: "45a6abe9796df8f05dc9686f3f307e98"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.094288 28490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "51d99dcff77f403b8fa9a08db02ca0b4"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "8892820ebf29d53910a8594c3952db9f"
server_key_iv: "45a6abe9796df8f05dc9686f3f307e98"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.100139 28490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 00:54:59.121738 28507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.123531 28490 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 00:54:59.123661 28490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "51d99dcff77f403b8fa9a08db02ca0b4"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "8892820ebf29d53910a8594c3952db9f"
server_key_iv: "45a6abe9796df8f05dc9686f3f307e98"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.123790 28490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:59.146895 28490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:59.151083 28490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:59.151373 28490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:59.152158 28490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:59.153338 28490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:59.153442 28490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.153584 28490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:59.153666 28490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.164413 28490 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:38719
I20260925 00:54:59.166416 28490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:54:59.169983 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 28490
I20260925 00:54:59.170112 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:54:59.170403 17192 external_mini_cluster.cc:1428] Setting key a2b8a8249503ff133a8273661378f1b5
I20260925 00:54:59.172346 28622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:38719 every 8 connection(s)
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:54:59.197880 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.174671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33027 (local address 127.16.202.1:50175)
0925 00:54:59.175184 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.175243 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.176583 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.180193 (+ 3610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.180204 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.180592 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:59.181351 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.181372 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.182719 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.182723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.184166 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.184179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.184901 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.184905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.185134 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.185813 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:59.185851 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:59.187666 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:59.194313 (+ 6647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:59.194321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:59.194340 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:59.194717 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:59.195099 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:59.195102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:59.195104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:59.195314 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:59.195842 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:59.195851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:59.196210 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.196420 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.196826 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260925 00:54:59.199043 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.175822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:50175 (local address 127.16.202.62:33027)
0925 00:54:59.179659 (+ 3837us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.179672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.179691 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:54:59.179744 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.179748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.179802 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.181532 (+ 1730us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:59.181664 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.182543 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.184340 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.184699 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.185486 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.192086 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:59.192090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:59.192093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:59.192130 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:59.194136 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.194876 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.194880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.194881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.194981 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.195451 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.195454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.195456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.195702 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:59.198615 (+ 2913us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.198702 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.198860 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3744,"thread_start_us":120,"threads_started":1}
I20260925 00:54:59.200461 28623 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33027
I20260925 00:54:59.200770 28623 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:59.201460 28623 heartbeater.cc:511] Master 127.16.202.62:33027 requested a full tablet report, sending...
I20260925 00:54:59.203289 28409 ts_manager.cc:194] Registered new tserver with Master: 51d99dcff77f403b8fa9a08db02ca0b4 (127.16.202.1:38719)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:54:59.204974 28409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:50175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:54:59.223824 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33027
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 00:54:59.351317 28629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:59.351768 28629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:59.351888 28629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:59.356828 28629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:59.356899 28629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:59.356980 28629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:54:59.362969 28629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.202.20:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:59.364185 28629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:59.365412 28629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:59.375236 28636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:59.375278 28635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:59.375252 28638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:59.375834 28629 server_base.cc:1034] running on GCE node
I20260925 00:54:59.376264 28629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:59.376951 28629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:59.378116 28629 hybrid_clock.cc:648] HybridClock initialized: now 1790297699378086 us; error 42 us; skew 500 ppm
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:59.380890 28629 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:54:59.381829 28629 webserver.cc:460] Webserver started at http://127.16.202.2:42477/ using document root <none> and password file <none>
I20260925 00:54:59.382372 28629 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:59.382431 28629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:59.382623 28629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:54:59.397324 28629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "733a98b4a9fe4e99a3e2d17329a9df6f"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "e3c702271b8c5135e465dde4d08a5d95"
server_key_iv: "f1e91e28bba64c9df8369fb356d8d5e4"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.398172 28629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "733a98b4a9fe4e99a3e2d17329a9df6f"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "e3c702271b8c5135e465dde4d08a5d95"
server_key_iv: "f1e91e28bba64c9df8369fb356d8d5e4"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.409286 28629 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260925 00:54:59.431375 28645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.433185 28629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:54:59.433336 28629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "733a98b4a9fe4e99a3e2d17329a9df6f"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "e3c702271b8c5135e465dde4d08a5d95"
server_key_iv: "f1e91e28bba64c9df8369fb356d8d5e4"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.433478 28629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:59.466562 28629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:59.470525 28629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:59.470788 28629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:59.471570 28629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:59.472728 28629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:59.472826 28629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.472920 28629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:59.472990 28629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.483271 28629 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:33247
I20260925 00:54:59.483335 28758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:33247 every 8 connection(s)
I20260925 00:54:59.484603 28629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:54:59.494125 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 28629
I20260925 00:54:59.494274 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:54:59.494593 17192 external_mini_cluster.cc:1428] Setting key c9ed280d31a67b1fce4ff7cefaa077bf
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:54:59.504971 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.488263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:60211 (local address 127.16.202.62:33027)
0925 00:54:59.488408 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.488413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.489539 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 00:54:59.490835 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.490839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.490897 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.490951 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:59.492870 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.493746 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.496939 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.497326 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.497406 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.500499 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:59.500504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:59.500507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:59.500538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:59.502301 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.502998 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.503002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.503004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.503114 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.503555 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.503559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.503560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.503792 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:59.503873 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.504632 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.504801 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:54:59.506659 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.488784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33027 (local address 127.16.202.2:60211)
0925 00:54:59.489361 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.489424 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.490559 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.491393 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.491403 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.491819 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:59.492661 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.492707 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.495109 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.495112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.496794 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.496806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.497613 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.497617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.497838 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.498539 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:59.498562 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:59.500272 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:59.502457 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:59.502464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:59.502485 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:59.502871 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:59.503233 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:59.503237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:59.503239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:59.503440 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:59.503942 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:59.503950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:59.504322 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.505190 (+ 868us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.505563 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":165,"threads_started":1}
I20260925 00:54:59.508216 28759 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33027
I20260925 00:54:59.508477 28759 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:59.509114 28759 heartbeater.cc:511] Master 127.16.202.62:33027 requested a full tablet report, sending...
I20260925 00:54:59.510275 28409 ts_manager.cc:194] Registered new tserver with Master: 733a98b4a9fe4e99a3e2d17329a9df6f (127.16.202.2:33247)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:54:59.511022 28409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:60211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:54:59.535663 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33027
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
W20260925 00:54:59.694936 28764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:54:59.695322 28764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:54:59.695389 28764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:54:59.701494 28764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:54:59.701678 28764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:54:59.701844 28764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:54:59.709753 28764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.202.20:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:54:59.711201 28764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:54:59.712510 28764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:54:59.724390 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:59.724577 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:54:59.724823 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:54:59.725814 28764 server_base.cc:1034] running on GCE node
I20260925 00:54:59.726321 28764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:54:59.727083 28764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:54:59.728566 28764 hybrid_clock.cc:648] HybridClock initialized: now 1790297699728496 us; error 82 us; skew 500 ppm
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:54:59.731935 28764 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:54:59.733040 28764 webserver.cc:460] Webserver started at http://127.16.202.3:34987/ using document root <none> and password file <none>
I20260925 00:54:59.733695 28764 fs_manager.cc:359] Metadata directory not provided
I20260925 00:54:59.733762 28764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:54:59.733963 28764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:54:59.746986 28764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "3f12d2c222024d5a91ede279f9dfe99f"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "e3ea5a3d22aeecef84e955a7b01035ca"
server_key_iv: "3eff2b25ec4b22de23a260bff710aa28"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.747830 28764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "3f12d2c222024d5a91ede279f9dfe99f"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "e3ea5a3d22aeecef84e955a7b01035ca"
server_key_iv: "3eff2b25ec4b22de23a260bff710aa28"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.753520 28764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 00:54:59.771262 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.773200 28764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:54:59.773341 28764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "3f12d2c222024d5a91ede279f9dfe99f"
format_stamp: "Formatted at 2026-09-25 00:54:59 on dist-test-slave-rpk1"
server_key: "e3ea5a3d22aeecef84e955a7b01035ca"
server_key_iv: "3eff2b25ec4b22de23a260bff710aa28"
server_key_version: "kuduclusterkey@0"
I20260925 00:54:59.773499 28764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:54:59.795116 28764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:54:59.798038 28764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:54:59.798213 28764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:54:59.798811 28764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:54:59.799644 28764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:54:59.799691 28764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.799743 28764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:54:59.799769 28764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:54:59.810061 28764 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:33321
I20260925 00:54:59.810148 28892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:33321 every 8 connection(s)
I20260925 00:54:59.811112 28764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:54:59.821231 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 28764
I20260925 00:54:59.821357 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:54:59.821681 17192 external_mini_cluster.cc:1428] Setting key c9c070170884c6c5aec37f8d9a3a1fe0
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:54:59.826845 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.812910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:42683 (local address 127.16.202.62:33027)
0925 00:54:59.814535 (+ 1625us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.814539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.814557 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:54:59.815395 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.815399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.815455 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.815550 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:59.816958 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.817767 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.819382 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.819740 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.819846 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.822668 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:59.822672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:59.822675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:59.822721 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:59.824367 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.825052 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.825056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.825058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.825183 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.825600 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.825603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.825605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.825823 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:59.825916 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.826569 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.826689 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533}
I20260925 00:54:59.828522 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.813468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:33027 (local address 127.16.202.3:42683)
0925 00:54:59.814042 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.814102 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.815141 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.815760 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.815770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.816122 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:54:59.816805 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.816823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.817901 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.817905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.819250 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.819261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.819904 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.819908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.820113 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.820801 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:54:59.820821 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:54:59.822464 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:54:59.824514 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:59.824521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:59.824540 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:59.824933 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:59.825295 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:54:59.825297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:54:59.825299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:54:59.825495 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:54:59.825981 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:54:59.825988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:54:59.826298 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.827031 (+ 733us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.827396 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260925 00:54:59.829882 28893 heartbeater.cc:347] Connected to a master server at 127.16.202.62:33027
I20260925 00:54:59.830154 28893 heartbeater.cc:464] Registering TS with master...
I20260925 00:54:59.830816 28893 heartbeater.cc:511] Master 127.16.202.62:33027 requested a full tablet report, sending...
I20260925 00:54:59.832051 28409 ts_manager.cc:194] Registered new tserver with Master: 3f12d2c222024d5a91ede279f9dfe99f (127.16.202.3:33321)
I20260925 00:54:59.832903 28409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:42683
I20260925 00:54:59.835664 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:54:59.847179 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.838284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.16.202.62:33027)
0925 00:54:59.838396 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.838400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.838414 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:54:59.838554 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.838557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.838602 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.838677 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:54:59.839594 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.840410 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.841604 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.841914 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.842017 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.843696 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:54:59.843700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:54:59.843703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:54:59.843727 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:54:59.845484 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.845957 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.845960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.845963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.846052 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:54:59.846333 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:54:59.846336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:54:59.846338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:54:59.846535 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:54:59.846649 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.846897 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.847017 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:54:59.850741 28409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:54:59.854697 28409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:54:59.875842 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.866591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33247 (local address 127.0.0.1:49124)
0925 00:54:59.867364 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.867378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.867468 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.869924 (+ 2456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.869927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.869943 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:59.870052 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.870058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.872217 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.872220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.873522 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.873533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.875302 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.875305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.875428 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.875474 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.875567 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.875691 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":49,"thread_start_us":84,"threads_started":1}
I20260925 00:54:59.876495 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.866873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:38719 (local address 127.0.0.1:56538)
0925 00:54:59.867647 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.867659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.867733 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.870194 (+ 2461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.870197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.870216 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:59.870309 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.870315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.871678 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.871682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.874493 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.874505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.875826 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.875830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.876194 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.876220 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.876292 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.876361 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":1404,"thread_start_us":82,"threads_started":1}
I20260925 00:54:59.876644 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.866998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.16.202.2:33247)
0925 00:54:59.869514 (+ 2516us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.869524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.869547 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:54:59.869613 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.869618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.869791 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.870185 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:59.870305 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.872100 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.873673 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.874466 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.876219 (+ 1753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.876363 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.876412 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.876512 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2412,"thread_start_us":123,"threads_started":1}
I20260925 00:54:59.880039 28693 tablet_service.cc:1467] Processing CreateTablet for tablet 40722d4ff0b54dd7bf97623a6f90ee02 (DEFAULT_TABLE table=test-table [id=b4318314f15242289eca9f564e4cfbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:54:59.881312 28693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40722d4ff0b54dd7bf97623a6f90ee02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:59.885936 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.867359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.16.202.1:38719)
0925 00:54:59.868411 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.868418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.868440 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:54:59.868900 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.868906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.869110 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.869241 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:59.870437 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.871546 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.874638 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.875494 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.875606 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.875764 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.884820 (+ 9056us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.885787 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":67,"threads_started":1}
I20260925 00:54:59.887516 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.865243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:33321 (local address 127.0.0.1:59342)
0925 00:54:59.865637 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.865690 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.865809 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.868522 (+ 2713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.868528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.868553 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:59.868728 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.868735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.872448 (+ 3713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.872452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.873679 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.873690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.887019 (+ 13329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.887024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.887158 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.887188 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.887276 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.887367 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260925 00:54:59.888487 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.865401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.16.202.3:33321)
0925 00:54:59.867978 (+ 2577us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.867994 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.868019 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:54:59.868117 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.868122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.868363 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.868535 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:59.868847 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.870066 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.877046 (+ 6980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.886863 (+ 9817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.887019 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.887162 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.888236 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.888348 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2466,"thread_start_us":139,"threads_started":1}
I20260925 00:54:59.890281 28557 tablet_service.cc:1467] Processing CreateTablet for tablet 40722d4ff0b54dd7bf97623a6f90ee02 (DEFAULT_TABLE table=test-table [id=b4318314f15242289eca9f564e4cfbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:54:59.891426 28557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40722d4ff0b54dd7bf97623a6f90ee02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:59.891812 28911 tablet_bootstrap.cc:492] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Bootstrap starting.
I20260925 00:54:59.891623 28827 tablet_service.cc:1467] Processing CreateTablet for tablet 40722d4ff0b54dd7bf97623a6f90ee02 (DEFAULT_TABLE table=test-table [id=b4318314f15242289eca9f564e4cfbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:54:59.892665 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40722d4ff0b54dd7bf97623a6f90ee02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:54:59.894642 28911 tablet_bootstrap.cc:654] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:59.895924 28911 log.cc:824] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:59.898073 28911 tablet_bootstrap.cc:492] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: No bootstrap required, opened a new log
I20260925 00:54:59.898278 28911 ts_tablet_manager.cc:1397] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:54:59.902614 28913 tablet_bootstrap.cc:492] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4: Bootstrap starting.
I20260925 00:54:59.902562 28911 raft_consensus.cc:348] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.902999 28911 raft_consensus.cc:374] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:59.903124 28911 raft_consensus.cc:729] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733a98b4a9fe4e99a3e2d17329a9df6f, State: Initialized, Role: FOLLOWER
I20260925 00:54:59.903700 28911 consensus_queue.cc:260] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.904897 28911 ts_tablet_manager.cc:1428] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 00:54:59.905223 28913 tablet_bootstrap.cc:654] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:59.905364 28759 heartbeater.cc:503] Master 127.16.202.62:33027 was elected leader, sending a full tablet report...
I20260925 00:54:59.906424 28913 log.cc:824] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:59.907140 28914 tablet_bootstrap.cc:492] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Bootstrap starting.
I20260925 00:54:59.910022 28914 tablet_bootstrap.cc:654] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Neither blocks nor log segments found. Creating new log.
I20260925 00:54:59.910991 28914 log.cc:824] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:54:59.913862 28914 tablet_bootstrap.cc:492] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: No bootstrap required, opened a new log
I20260925 00:54:59.914086 28914 ts_tablet_manager.cc:1397] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:54:59.918211 28914 raft_consensus.cc:348] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.918534 28914 raft_consensus.cc:374] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:59.918586 28914 raft_consensus.cc:729] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f12d2c222024d5a91ede279f9dfe99f, State: Initialized, Role: FOLLOWER
I20260925 00:54:59.919090 28914 consensus_queue.cc:260] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.920203 28914 ts_tablet_manager.cc:1428] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:54:59.920632 28893 heartbeater.cc:503] Master 127.16.202.62:33027 was elected leader, sending a full tablet report...
I20260925 00:54:59.923255 28913 tablet_bootstrap.cc:492] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4: No bootstrap required, opened a new log
I20260925 00:54:59.923895 28913 ts_tablet_manager.cc:1397] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260925 00:54:59.927395 28913 raft_consensus.cc:348] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.927652 28913 raft_consensus.cc:374] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:54:59.927734 28913 raft_consensus.cc:729] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d99dcff77f403b8fa9a08db02ca0b4, State: Initialized, Role: FOLLOWER
I20260925 00:54:59.928197 28913 consensus_queue.cc:260] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.929265 28913 ts_tablet_manager.cc:1428] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:54:59.929633 28623 heartbeater.cc:503] Master 127.16.202.62:33027 was elected leader, sending a full tablet report...
I20260925 00:54:59.963897 28919 raft_consensus.cc:482] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:54:59.964550 28919 raft_consensus.cc:504] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.967381 28919 leader_election.cc:288] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f12d2c222024d5a91ede279f9dfe99f (127.16.202.3:33321), 733a98b4a9fe4e99a3e2d17329a9df6f (127.16.202.2:33247)
I20260925 00:54:59.971975 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.966466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:40503 (local address 127.16.202.3:33321)
0925 00:54:59.966616 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.966621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.967778 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 00:54:59.967987 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.967990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.968054 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.968105 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:59.968546 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.969153 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.970674 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.971291 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.971382 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.971432 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.971741 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.971815 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:54:59.972657 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.966386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:33321 (local address 127.16.202.1:40503)
0925 00:54:59.967688 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.967705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.967860 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.968253 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.968255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.968275 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:59.968411 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.968418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.969299 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.969302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.970542 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.970552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.971530 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.971534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.971622 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.971643 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.972423 (+ 780us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.972499 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"thread_start_us":117,"threads_started":1}
I20260925 00:54:59.973122 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.967237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:51753 (local address 127.16.202.2:33247)
0925 00:54:59.967350 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:54:59.967354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:54:59.969283 (+ 1929us) server_negotiation.cc:407] Connection header received
0925 00:54:59.969423 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:54:59.969426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:54:59.969477 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:54:59.969522 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:54:59.969926 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.970638 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.971939 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.972456 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.972555 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.972600 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 00:54:59.972938 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 00:54:59.973016 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:54:59.973481 28847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40722d4ff0b54dd7bf97623a6f90ee02" candidate_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f12d2c222024d5a91ede279f9dfe99f" is_pre_election: true
I20260925 00:54:59.973649 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:54:59.966889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33247 (local address 127.16.202.1:51753)
0925 00:54:59.968699 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0925 00:54:59.968784 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 00:54:59.969297 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:54:59.969663 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:54:59.969666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:54:59.969682 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:54:59.969803 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.969809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.970775 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.970778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.971821 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:54:59.971829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:54:59.972707 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:54:59.972710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:54:59.972807 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:54:59.972827 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:54:59.973430 (+ 603us) client_negotiation.cc:240] Negotiation successful
0925 00:54:59.973500 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"mutex_wait_us":11,"thread_start_us":82,"threads_started":1}
I20260925 00:54:59.973858 28847 raft_consensus.cc:2454] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51d99dcff77f403b8fa9a08db02ca0b4 in term 0.
I20260925 00:54:59.974195 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40722d4ff0b54dd7bf97623a6f90ee02" candidate_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" is_pre_election: true
I20260925 00:54:59.974475 28711 raft_consensus.cc:2454] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51d99dcff77f403b8fa9a08db02ca0b4 in term 0.
I20260925 00:54:59.974587 28510 leader_election.cc:302] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f12d2c222024d5a91ede279f9dfe99f, 51d99dcff77f403b8fa9a08db02ca0b4; no voters:
I20260925 00:54:59.974979 28919 raft_consensus.cc:2781] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:54:59.975072 28919 raft_consensus.cc:482] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:54:59.975116 28919 raft_consensus.cc:3037] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:59.976383 28919 raft_consensus.cc:504] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.976755 28919 leader_election.cc:288] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [CANDIDATE]: Term 1 election: Requested vote from peers 3f12d2c222024d5a91ede279f9dfe99f (127.16.202.3:33321), 733a98b4a9fe4e99a3e2d17329a9df6f (127.16.202.2:33247)
I20260925 00:54:59.977141 28847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40722d4ff0b54dd7bf97623a6f90ee02" candidate_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f12d2c222024d5a91ede279f9dfe99f"
I20260925 00:54:59.977140 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40722d4ff0b54dd7bf97623a6f90ee02" candidate_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f"
I20260925 00:54:59.977259 28847 raft_consensus.cc:3037] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:59.977259 28711 raft_consensus.cc:3037] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:54:59.978502 28847 raft_consensus.cc:2454] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d99dcff77f403b8fa9a08db02ca0b4 in term 1.
I20260925 00:54:59.978500 28711 raft_consensus.cc:2454] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d99dcff77f403b8fa9a08db02ca0b4 in term 1.
I20260925 00:54:59.978828 28510 leader_election.cc:302] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f12d2c222024d5a91ede279f9dfe99f, 51d99dcff77f403b8fa9a08db02ca0b4; no voters:
I20260925 00:54:59.979275 28919 raft_consensus.cc:2781] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:54:59.979626 28919 raft_consensus.cc:686] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [term 1 LEADER]: Becoming Leader. State: Replica: 51d99dcff77f403b8fa9a08db02ca0b4, State: Running, Role: LEADER
I20260925 00:54:59.980261 28919 consensus_queue.cc:237] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } }
I20260925 00:54:59.983686 28408 catalog_manager.cc:5495] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51d99dcff77f403b8fa9a08db02ca0b4 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51d99dcff77f403b8fa9a08db02ca0b4" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 38719 } health_report { overall_health: HEALTHY } } }
W20260925 00:54:59.987030 28760 tablet.cc:2369] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 00:54:59 dist-test-slave-rpk1 krb5kdc[27933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297699, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:00.005587 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:00.000416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.16.202.1:38719)
0925 00:55:00.000528 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:00.000544 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:00.000556 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:55:00.000637 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:00.000640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:00.000723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:00.000820 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:55:00.001176 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:00.001953 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:00.003328 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:00.003676 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:00.003768 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:00.004196 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:55:00.004792 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:55:00.004902 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:00.005360 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 00:55:00.005428 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:55:00.013482 28711 raft_consensus.cc:1261] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 1 FOLLOWER]: Refusing update from remote peer 51d99dcff77f403b8fa9a08db02ca0b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:00.013482 28847 raft_consensus.cc:1261] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 1 FOLLOWER]: Refusing update from remote peer 51d99dcff77f403b8fa9a08db02ca0b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:00.014343 28919 consensus_queue.cc:1035] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733a98b4a9fe4e99a3e2d17329a9df6f" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:00.014838 28919 consensus_queue.cc:1035] T 40722d4ff0b54dd7bf97623a6f90ee02 P 51d99dcff77f403b8fa9a08db02ca0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f12d2c222024d5a91ede279f9dfe99f" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:00.037295 28928 mvcc.cc:205] Tried to move back new op lower bound from 7333059379244855296 to 7333059379122413568. Current Snapshot: MvccSnapshot[applied={T|T < 7333059379244855296}]
I20260925 00:55:00.054919 28408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:00.055135 28408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:00.059353 28408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 622d1db84611452ab50c788780953ed8: Sending DeleteTablet for 3 replicas of tablet 40722d4ff0b54dd7bf97623a6f90ee02
I20260925 00:55:00.060480 28827 tablet_service.cc:1514] Processing DeleteTablet for tablet 40722d4ff0b54dd7bf97623a6f90ee02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:00 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:59342
I20260925 00:55:00.061061 28557 tablet_service.cc:1514] Processing DeleteTablet for tablet 40722d4ff0b54dd7bf97623a6f90ee02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:00 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:56538
I20260925 00:55:00.061513 28693 tablet_service.cc:1514] Processing DeleteTablet for tablet 40722d4ff0b54dd7bf97623a6f90ee02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:00 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49124
I20260925 00:55:00.063680 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 28490
I20260925 00:55:00.066097 28939 tablet_replica.cc:330] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: stopping tablet replica
I20260925 00:55:00.066465 28939 raft_consensus.cc:2229] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:00.066838 28939 raft_consensus.cc:2258] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:00.067335 28937 tablet_replica.cc:330] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: stopping tablet replica
I20260925 00:55:00.067660 28937 raft_consensus.cc:2229] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:00.067955 28937 raft_consensus.cc:2258] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:00.069831 28939 ts_tablet_manager.cc:1905] T 40722d4ff0b54dd7bf97623a6f90ee02 P 733a98b4a9fe4e99a3e2d17329a9df6f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:55:00.071287 28393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:38719 (error 108)
I20260925 00:55:00.072902 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 28629
I20260925 00:55:00.074489 28937 ts_tablet_manager.cc:1905] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:55:00.077806 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:00.072619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:38719 (local address 127.0.0.1:56560)
0925 00:55:00.077582 (+ 4963us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:00.077693 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:38719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4841}
W20260925 00:55:00.078076 28393 catalog_manager.cc:4584] TS 51d99dcff77f403b8fa9a08db02ca0b4 (127.16.202.1:38719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40722d4ff0b54dd7bf97623a6f90ee02: Network error: Client connection negotiation failed: client connection to 127.16.202.1:38719: connect: Connection refused (error 111)
I20260925 00:55:00.082602 28937 ts_tablet_manager.cc:1918] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:00.082753 28937 log.cc:1196] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/40722d4ff0b54dd7bf97623a6f90ee02
I20260925 00:55:00.083104 28937 ts_tablet_manager.cc:1939] T 40722d4ff0b54dd7bf97623a6f90ee02 P 3f12d2c222024d5a91ede279f9dfe99f: Deleting consensus metadata
I20260925 00:55:00.084563 28395 catalog_manager.cc:4841] TS 3f12d2c222024d5a91ede279f9dfe99f (127.16.202.3:33321): tablet 40722d4ff0b54dd7bf97623a6f90ee02 (table test-table [id=b4318314f15242289eca9f564e4cfbbe]) successfully deleted
W20260925 00:55:00.084818 28905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:55:00.084191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33247 (local address 127.0.0.1:49134)
0925 00:55:00.084372 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:00.084387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:00.084475 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:00.084714 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.2:33247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 00:55:00.085371 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 28764
I20260925 00:55:00.092303 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 28377
I20260925 00:55:00.100044 17192 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 00:55:00.435462 17192 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 00:55:00.435613 17192 mini_ranger.cc:66] Stopping Ranger...
I20260925 00:55:00.771018 17192 mini_ranger.cc:68] Stopped Ranger
2026-09-25 00:55:00.771 UTC [27963] LOG: received smart shutdown request
2026-09-25 00:55:00.774 UTC [27963] LOG: background worker "logical replication launcher" (PID 27972) exited with exit code 1
2026-09-25 00:55:00.774 UTC [27967] LOG: shutting down
2026-09-25 00:55:00.948 UTC [27963] LOG: database system is shut down
2026-09-25T00:55:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46854 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:55:01 dist-test-slave-rpk1 krb5kdc[28949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:55:01 dist-test-slave-rpk1 krb5kdc[28949](info): set up 2 sockets
Sep 25 00:55:01 dist-test-slave-rpk1 krb5kdc[28949](info): commencing operation
krb5kdc: starting...
W20260925 00:55:02.000837 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:55:02 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:55:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/rangeradmin_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/rangeradmin_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/rangerlookup_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/rangerlookup_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
I20260925 00:55:02.167327 17192 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/postgres -l logfile start
2026-09-25 00:55:04.689 UTC [28979] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 00:55:04.689 UTC [28979] LOG: listening on IPv4 address "127.16.202.20", port 51471
2026-09-25 00:55:04.693 UTC [28979] LOG: listening on Unix socket "/tmp/.s.PGSQL.51471"
2026-09-25 00:55:04.713 UTC [28982] LOG: database system was shut down at 2026-09-25 00:55:03 UTC
2026-09-25 00:55:04.717 UTC [28979] LOG: database system is ready to accept connections
I20260925 00:55:05.597226 17192 mini_postgres.cc:96] Postgres bound to 51471
2026-09-25 00:55:05.603 UTC [28990] FATAL: database "slave" does not exist
127.16.202.20:51471 - accepting connections
I20260925 00:55:05.604302 17192 mini_ranger.cc:161] Starting Ranger...
I20260925 00:55:05.619266 17192 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 00:55:06.150060 17192 mini_ranger.cc:87] Created ranger Postgres database
I20260925 00:55:06.150147 17192 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-admin
2026-09-25 00:55:06,419 [I] DB FLAVOR :POSTGRES
2026-09-25 00:55:06,419 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:55:06,419 [I] Checking connection..
2026-09-25 00:55:06,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 00:55:06,671 [I] Checking connection passed.
2026-09-25 00:55:06,672 [I] --------- Verifying version history table ---------
2026-09-25 00:55:06,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:55:06,935 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:55:06,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:55:07,192 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:55:07,193 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 00:55:07,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 00:55:07.443 UTC [29054] WARNING: there is no transaction in progress
2026-09-25 00:55:07,449 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 00:55:07,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:55:07,707 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 00:55:07,707 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:55:07,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 00:55:07,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 00:55:08,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rpk1.c.gcp-upstream.internal','N') ;"
2026-09-25 00:55:08,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:55:08,652 [I] Table x_portal_user does not exist in database ranger
2026-09-25 00:55:08,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:55:08,889 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 00:55:08,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:55:09,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:55:09,382 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 00:55:09,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 00:55:10.412 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.430 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.448 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.463 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.484 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.503 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.571 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.578 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.594 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.605 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.616 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.632 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.642 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.649 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.657 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:10.664 UTC [29162] WARNING: there is no transaction in progress
2026-09-25 00:55:11,411 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 00:55:11,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 00:55:11,647 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 00:55:11,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:55:11,916 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 00:55:11,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:55:12,165 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 00:55:12,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:55:12,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:55:12,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rpk1.c.gcp-upstream.internal';"
2026-09-25 00:55:12,937 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 00:55:12,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 00:55:13,172 [I] DB_PATCHES have already been applied
I20260925 00:55:13.178373 17192 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 00:55:13.178443 17192 mini_ranger.cc:193] Using host: 127.16.202.20
I20260925 00:55:13.181742 17192 mini_ranger.cc:227] Ranger admin URL: http://127.16.202.20:59271
Sep 25, 2026 12:55:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:55:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:55:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:55:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:55:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 00:55:14.193221 17192 mini_ranger.cc:160] Time spent starting Ranger: real 8.589s user 0.003s sys 0.006s
Sep 25, 2026 12:55:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:55:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 12:55:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:55:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:55:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 00:55:35.758311 17192 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.202.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.202.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/rangerkms_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.202.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/rangerkms_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.202.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.202.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.202.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/HTTP_127.16.202.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 00:55:35 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 00:55:35.848884 17192 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 00:55:35.858954 17192 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 00:55:36.807435 17192 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 00:55:36.807513 17192 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms
I20260925 00:55:36.807531 17192 mini_ranger_kms.cc:227] Using postgres at 127.16.202.20:51471
2026-09-25 00:55:36,962 [I] DB FLAVOR :POSTGRES
2026-09-25 00:55:36,962 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:55:36,962 [I] Checking connection
2026-09-25 00:55:36,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 00:55:37,196 [I] connection success
2026-09-25 00:55:37,196 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 00:55:37,196 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 00:55:37,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 00:55:37,452 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 00:55:37,452 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:55:37,453 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 00:55:37,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.202.20:51471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 00:55:37,709 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 00:55:38.195005 17192 mini_ranger_kms.cc:310] Created kms service
I20260925 00:55:38.358773 17192 mini_ranger_kms.cc:326] Created kudu user
I20260925 00:55:38.400234 17192 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 00:55:38 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:38.835057 17192 mini_ranger_kms.cc:384] Added ranger policy
I20260925 00:55:38.835223 17192 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiPY1ZZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiPY1ZZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 00:55:38.835245 17192 mini_ranger_kms.cc:240] Using host: 127.16.202.20
I20260925 00:55:38.839448 17192 mini_ranger_kms.cc:276] Ranger KMS PID: 29342
I20260925 00:55:38.839516 17192 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.202.20:43951
Sep 25, 2026 12:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 00:55:40.851294 17192 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.002s user 0.012s sys 0.126s
I20260925 00:55:40.851414 17192 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 00:55:40.851480 17192 mini_ranger_kms.cc:401] 127.16.202.20:43951/kms/v1/keys
Sep 25 00:55:41 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.202.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 00:55:41 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.202.20@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
Sep 25 00:55:42 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.202.20@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
Sep 25 00:55:44 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:55:44.376737 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36559
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36559
--ranger_config_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:44.486511 29393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:44.486826 29393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:44.486882 29393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:44.491083 29393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:44.491155 29393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:44.491182 29393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:44.491204 29393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:44.491223 29393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:44.496913 29393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.202.20:43951/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36559
--ranger_config_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:44.498008 29393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:44.499054 29393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:55:44.507994 29393 server_base.cc:1034] running on GCE node
W20260925 00:55:44.507926 29399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:44.507937 29398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:44.507936 29401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:44.508845 29393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:44.510007 29393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:44.511194 29393 hybrid_clock.cc:648] HybridClock initialized: now 1790297744511178 us; error 34 us; skew 500 ppm
Sep 25 00:55:44 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:44.517351 29393 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:44.518301 29393 webserver.cc:460] Webserver started at http://127.16.202.62:42271/ using document root <none> and password file <none>
I20260925 00:55:44.518821 29393 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:44.518878 29393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:44.519091 29393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:55:44 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297744, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:44.581182 29393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "778e64b6233f4a89bdcb6339525c5f89"
format_stamp: "Formatted at 2026-09-25 00:55:44 on dist-test-slave-rpk1"
server_key: "378fb879c0d38cfdbaa3fc289407e7ef"
server_key_iv: "23917d62abd329c9b5a09b82905c3304"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:44.581827 29393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "778e64b6233f4a89bdcb6339525c5f89"
format_stamp: "Formatted at 2026-09-25 00:55:44 on dist-test-slave-rpk1"
server_key: "378fb879c0d38cfdbaa3fc289407e7ef"
server_key_iv: "23917d62abd329c9b5a09b82905c3304"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:44.585958 29393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:55:44.604806 29408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:44.607285 29393 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 00:55:44.607622 29393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "778e64b6233f4a89bdcb6339525c5f89"
format_stamp: "Formatted at 2026-09-25 00:55:44 on dist-test-slave-rpk1"
server_key: "378fb879c0d38cfdbaa3fc289407e7ef"
server_key_iv: "23917d62abd329c9b5a09b82905c3304"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:44.607911 29393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:44.636035 29393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:44.639163 29393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:44.639346 29393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:44.646399 29393 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36559
I20260925 00:55:44.646421 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36559 every 8 connection(s)
I20260925 00:55:44.647363 29393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:44.650209 29461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:44.655294 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 29393
I20260925 00:55:44.655409 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:55:44.655669 17192 external_mini_cluster.cc:1428] Setting key 1da59253eaf9a6d79089d602be2dcdc5
I20260925 00:55:44.656036 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Bootstrap starting.
I20260925 00:55:44.658828 29461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:44.659749 29461 log.cc:824] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:44.662101 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: No bootstrap required, opened a new log
Sep 25 00:55:44 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:44.666481 29461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:44.666729 29461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:44.666793 29461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778e64b6233f4a89bdcb6339525c5f89, State: Initialized, Role: FOLLOWER
I20260925 00:55:44.667356 29461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:44.667521 29461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:55:44.667589 29461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:55:44.667689 29461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:44.668843 29461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:44.669248 29461 leader_election.cc:302] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778e64b6233f4a89bdcb6339525c5f89; no voters:
I20260925 00:55:44.669595 29461 leader_election.cc:288] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:55:44.669739 29466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:44.669950 29466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 LEADER]: Becoming Leader. State: Replica: 778e64b6233f4a89bdcb6339525c5f89, State: Running, Role: LEADER
I20260925 00:55:44.670353 29466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:44.670856 29461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:44.671675 29466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778e64b6233f4a89bdcb6339525c5f89. Latest consensus state: current_term: 1 leader_uuid: "778e64b6233f4a89bdcb6339525c5f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } } }
I20260925 00:55:44.671886 29466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:44.672112 29467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778e64b6233f4a89bdcb6339525c5f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } } }
I20260925 00:55:44.672175 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:44.657177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.16.202.62:36559)
0925 00:55:44.657672 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:44.657691 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:44.657737 (+ 46us) server_negotiation.cc:407] Connection header received
0925 00:55:44.658576 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:44.658601 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:44.659103 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:44.659482 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:44.660244 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:44.661795 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:44.662980 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:44.663464 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:44.663597 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:44.666369 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:44.666375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:44.666390 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:44.666419 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:44.668214 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:44.668792 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:44.668798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:44.668804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:44.668947 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:44.669289 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:44.669293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:44.669295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:44.669844 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:44.670666 (+ 822us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:44.670821 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 00:55:44.671141 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":142,"threads_started":1}
I20260925 00:55:44.672204 29467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:44.672303 29475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:44.675454 29475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:44.675894 29461 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 00:55:44.684634 29475 catalog_manager.cc:1347] Generated new cluster ID: 744f890061474f5c8b6cce8086edeb80
I20260925 00:55:44.684749 29475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:55:44.707932 29475 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:55:44.709082 29475 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:55:44.723066 29475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Generated new TSK 0
I20260925 00:55:44.724200 29475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:55:45.408387 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36559
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:45.562364 29506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:45.562796 29506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:45.562896 29506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:45.568729 29506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:45.568861 29506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:45.569005 29506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:55:45.575847 29506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.202.20:43951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:45.577028 29506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:45.578048 29506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:45.588438 29513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:45.588614 29514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:45.588486 29516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:45.588783 29506 server_base.cc:1034] running on GCE node
I20260925 00:55:45.589522 29506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:45.590286 29506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:45.591483 29506 hybrid_clock.cc:648] HybridClock initialized: now 1790297745591411 us; error 101 us; skew 500 ppm
Sep 25 00:55:45 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:45.595427 29506 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:55:45.596766 29506 webserver.cc:460] Webserver started at http://127.16.202.1:41243/ using document root <none> and password file <none>
I20260925 00:55:45.597608 29506 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:45.597702 29506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:45.597930 29506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:55:45 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:45.613279 29506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "975ed1b8c7de467ab2b40ff2d085d103"
format_stamp: "Formatted at 2026-09-25 00:55:45 on dist-test-slave-rpk1"
server_key: "f6f869e101ccdda49585e9a3776a87aa"
server_key_iv: "c605d4e41141dd306605ebcbca29c85a"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:45.616197 29506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "975ed1b8c7de467ab2b40ff2d085d103"
format_stamp: "Formatted at 2026-09-25 00:55:45 on dist-test-slave-rpk1"
server_key: "f6f869e101ccdda49585e9a3776a87aa"
server_key_iv: "c605d4e41141dd306605ebcbca29c85a"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:45.621970 29506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 00:55:45.653106 29523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:45.654974 29506 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.001s
I20260925 00:55:45.655123 29506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "975ed1b8c7de467ab2b40ff2d085d103"
format_stamp: "Formatted at 2026-09-25 00:55:45 on dist-test-slave-rpk1"
server_key: "f6f869e101ccdda49585e9a3776a87aa"
server_key_iv: "c605d4e41141dd306605ebcbca29c85a"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:45.655289 29506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:45.695986 29506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:45.699741 29506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:45.700102 29506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:45.702033 29506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:45.704334 29506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:45.704629 29506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 00:55:45.704843 29506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:45.704936 29506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:45.719761 29506 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:35199
I20260925 00:55:45.721185 29506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:55:45.729652 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 29506
I20260925 00:55:45.729809 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:55:45.730131 17192 external_mini_cluster.cc:1428] Setting key dcd243cb2be6f78ebfafc3895d40ad80
Sep 25 00:55:45 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:45.740238 29638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:35199 every 8 connection(s)
I20260925 00:55:45.745035 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:45.724898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:54425 (local address 127.16.202.62:36559)
0925 00:55:45.725140 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:45.725145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:45.730102 (+ 4957us) server_negotiation.cc:407] Connection header received
0925 00:55:45.731937 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:45.731941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:45.732055 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:45.732199 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:45.734145 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:45.734906 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:45.736764 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:45.737100 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:45.737187 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:45.740393 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:45.740397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:45.740399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:45.740431 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:45.742244 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:45.742919 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:45.742923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:45.742925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:45.743043 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:45.743468 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:45.743471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:45.743473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:45.743725 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:45.743811 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:45.744614 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 00:55:45.744813 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260925 00:55:45.746840 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:45.723893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36559 (local address 127.16.202.1:54425)
0925 00:55:45.729766 (+ 5873us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:45.729961 (+ 195us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:45.731677 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:45.732624 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:45.732636 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:45.733090 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:45.733913 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:45.733939 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:45.735088 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:45.735092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:45.736566 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:45.736579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:45.737418 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:45.737421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:45.737639 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:45.738338 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:45.738364 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:45.740157 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:45.742383 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:45.742391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:45.742411 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:45.742787 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:45.743158 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:45.743161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:45.743164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:45.743354 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:45.743882 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:45.743890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:45.744264 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 00:55:45.745257 (+ 993us) client_negotiation.cc:240] Negotiation successful
0925 00:55:45.745646 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5524,"thread_start_us":136,"threads_started":1}
I20260925 00:55:45.749089 29639 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36559
I20260925 00:55:45.749368 29639 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:45.750008 29639 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:45.751993 29425 ts_manager.cc:194] Registered new tserver with Master: 975ed1b8c7de467ab2b40ff2d085d103 (127.16.202.1:35199)
I20260925 00:55:45.753620 29425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:54425
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
Sep 25 00:55:45 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:55:45.780460 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36559
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:45.940145 29645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:45.940550 29645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:45.940616 29645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:45.947468 29645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:45.947574 29645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:45.947664 29645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:55:45.956558 29645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.202.20:43951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:45.958148 29645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:45.960021 29645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:45.974627 29654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:45.976001 29645 server_base.cc:1034] running on GCE node
W20260925 00:55:45.976279 29651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:45.977411 29652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:45.978070 29645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:45.978978 29645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:45.982285 29645 hybrid_clock.cc:648] HybridClock initialized: now 1790297745982239 us; error 67 us; skew 500 ppm
Sep 25 00:55:45 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:45.986205 29645 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:55:45.987473 29645 webserver.cc:460] Webserver started at http://127.16.202.2:37855/ using document root <none> and password file <none>
I20260925 00:55:45.988318 29645 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:45.988478 29645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:45.988808 29645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:55:45 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:46.006675 29645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "141c38713e384e40b510e56849a85f5e"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "a76024ddef66d173f01d19a819b0198a"
server_key_iv: "5e8431387d94561dcc7a81a02cd09aee"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.007479 29645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "141c38713e384e40b510e56849a85f5e"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "a76024ddef66d173f01d19a819b0198a"
server_key_iv: "5e8431387d94561dcc7a81a02cd09aee"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.012502 29645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 00:55:46.029693 29661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.031637 29645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:55:46.031795 29645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "141c38713e384e40b510e56849a85f5e"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "a76024ddef66d173f01d19a819b0198a"
server_key_iv: "5e8431387d94561dcc7a81a02cd09aee"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.031949 29645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:46.063226 29645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:46.067072 29645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:46.067279 29645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:46.068028 29645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:46.069123 29645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:46.069192 29645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.069252 29645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:46.069280 29645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.079855 29645 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:35807
I20260925 00:55:46.081139 29645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:55:46.081805 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 29645
I20260925 00:55:46.081928 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:55:46.082245 17192 external_mini_cluster.cc:1428] Setting key 8d4a0ef7c54cfb59da373382339a33a0
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:55:46.095225 29774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:35807 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:46.123942 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:46.100925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:32769 (local address 127.16.202.62:36559)
0925 00:55:46.101063 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:46.101068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:46.103104 (+ 2036us) server_negotiation.cc:407] Connection header received
0925 00:55:46.104440 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:46.104444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:46.104506 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:46.104563 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:46.106470 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.107307 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.112344 (+ 5037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.112766 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.112885 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:46.119175 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:46.119181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:46.119184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:46.119221 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:46.121128 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:46.121938 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:46.121942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:46.121944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:46.122047 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:46.122505 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:46.122508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:46.122510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:46.122752 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:46.122839 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:46.123609 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 00:55:46.123759 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:55:46.125787 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:46.101509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36559 (local address 127.16.202.2:32769)
0925 00:55:46.102897 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:46.102978 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:46.104152 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:46.105067 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:46.105077 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:46.105494 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:46.106265 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:46.106287 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.107488 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.107492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:46.112122 (+ 4630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:46.112146 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.113175 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.113180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:46.113422 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:46.116931 (+ 3509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:46.116961 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:46.118860 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:46.121326 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:46.121334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:46.121357 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:46.121785 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:46.122168 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:46.122172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:46.122174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:46.122388 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:46.122914 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:46.122934 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:46.123308 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 00:55:46.124155 (+ 847us) client_negotiation.cc:240] Negotiation successful
0925 00:55:46.124547 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":181,"threads_started":1}
I20260925 00:55:46.127350 29775 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36559
I20260925 00:55:46.127668 29775 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:46.128324 29775 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:46.129766 29425 ts_manager.cc:194] Registered new tserver with Master: 141c38713e384e40b510e56849a85f5e (127.16.202.2:35807)
I20260925 00:55:46.130558 29425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:32769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:55:46.146551 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36559
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
W20260925 00:55:46.319407 29780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:46.320256 29780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:46.320386 29780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:46.326900 29780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:46.327071 29780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:46.327198 29780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:55:46.335027 29780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.202.20:43951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:46.336630 29780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:46.338029 29780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:46.351789 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:46.352407 29785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:46.351789 29788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:46.353681 29780 server_base.cc:1034] running on GCE node
I20260925 00:55:46.354319 29780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:46.355154 29780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:46.356448 29780 hybrid_clock.cc:648] HybridClock initialized: now 1790297746356384 us; error 85 us; skew 500 ppm
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:46.360180 29780 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:55:46.361316 29780 webserver.cc:460] Webserver started at http://127.16.202.3:46115/ using document root <none> and password file <none>
I20260925 00:55:46.362010 29780 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:46.362077 29780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:46.362294 29780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:46.388253 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "b8b09cae577143948e6873593b411306"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "b0b807c47d1e36167acd73caa4677d61"
server_key_iv: "eaf4543d88a1bd1978d2f2eab16efeb2"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.389158 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "b8b09cae577143948e6873593b411306"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "b0b807c47d1e36167acd73caa4677d61"
server_key_iv: "eaf4543d88a1bd1978d2f2eab16efeb2"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.402557 29780 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260925 00:55:46.422389 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.424178 29780 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:55:46.424321 29780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "b8b09cae577143948e6873593b411306"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "b0b807c47d1e36167acd73caa4677d61"
server_key_iv: "eaf4543d88a1bd1978d2f2eab16efeb2"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.424476 29780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:46.438218 29780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:46.442070 29780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:46.442373 29780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:46.443200 29780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:46.444375 29780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:46.444553 29780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.444659 29780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:46.444747 29780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.455085 29780 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:34465
I20260925 00:55:46.456324 29780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:55:46.460000 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 29780
I20260925 00:55:46.460131 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:55:46.460490 17192 external_mini_cluster.cc:1428] Setting key 9a922dee57341c3c50e759e08e4d574b
I20260925 00:55:46.465770 29908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:34465 every 8 connection(s)
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:46.485874 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:46.467895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:45425 (local address 127.16.202.62:36559)
0925 00:55:46.468045 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:46.468050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:46.470967 (+ 2917us) server_negotiation.cc:407] Connection header received
0925 00:55:46.473258 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:46.473262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:46.473322 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:46.473373 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:46.475178 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.476047 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.477671 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.477981 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.478054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:46.481322 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:46.481326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:46.481329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:46.481356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:46.483081 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:46.483835 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:46.483839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:46.483841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:46.483953 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:46.484406 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:46.484409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:46.484411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:46.484666 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:46.484781 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:46.485550 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 00:55:46.485696 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:55:46.487972 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:46.468581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36559 (local address 127.16.202.3:45425)
0925 00:55:46.470785 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:46.470857 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:46.472969 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:46.473838 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:46.473849 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:46.474243 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:46.474982 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:46.475007 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.476215 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.476219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:46.477546 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:46.477558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:46.478259 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:46.478264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:46.478476 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:46.479243 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:46.479266 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:46.481109 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:46.483287 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:46.483294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:46.483314 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:46.483709 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:46.484070 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:46.484073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:46.484075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:46.484284 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:46.484851 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:46.484859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:46.485255 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 00:55:46.486401 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0925 00:55:46.486818 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"thread_start_us":160,"threads_started":1}
I20260925 00:55:46.489946 29909 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36559
I20260925 00:55:46.490239 29909 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:46.490900 29909 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:46.492563 29420 ts_manager.cc:194] Registered new tserver with Master: b8b09cae577143948e6873593b411306 (127.16.202.3:34465)
I20260925 00:55:46.493419 29420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:45425
I20260925 00:55:46.494736 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:55:46.495471 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 29506
I20260925 00:55:46.507623 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 29645
I20260925 00:55:46.515938 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 29780
I20260925 00:55:46.522225 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 29393
I20260925 00:55:46.528362 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36559
--webserver_interface=127.16.202.62
--webserver_port=42271
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36559
--ranger_config_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:46.641312 29912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:46.641657 29912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:46.641726 29912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:46.646284 29912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:46.646382 29912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:46.646413 29912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:46.646435 29912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:46.646456 29912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:46.652361 29912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36559
--ranger_config_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=42271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:46.653599 29912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:46.654701 29912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:46.663445 29917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:46.663504 29920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:46.663450 29918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:46.663991 29912 server_base.cc:1034] running on GCE node
I20260925 00:55:46.664603 29912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:46.665917 29912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:46.667132 29912 hybrid_clock.cc:648] HybridClock initialized: now 1790297746667108 us; error 47 us; skew 500 ppm
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:46.669945 29912 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:46.670967 29912 webserver.cc:460] Webserver started at http://127.16.202.62:42271/ using document root <none> and password file <none>
I20260925 00:55:46.671525 29912 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:46.671597 29912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:46.721786 29912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:46.724030 29928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:46.725469 29912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:46.725607 29912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "778e64b6233f4a89bdcb6339525c5f89"
format_stamp: "Formatted at 2026-09-25 00:55:44 on dist-test-slave-rpk1"
server_key: "378fb879c0d38cfdbaa3fc289407e7ef"
server_key_iv: "23917d62abd329c9b5a09b82905c3304"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:46.726203 29912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:46.745810 29912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:46.748899 29912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:46.749089 29912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:46.756358 29912 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36559
I20260925 00:55:46.756381 29980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36559 every 8 connection(s)
I20260925 00:55:46.757532 29912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:46.762006 29981 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:55:46.764148 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Bootstrap starting.
I20260925 00:55:46.765803 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 29912
I20260925 00:55:46.766419 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:35199
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:36559
--webserver_port=41243
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:46.780932 29981 log.cc:824] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:46.787665 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:55:46.788043 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Bootstrap complete.
I20260925 00:55:46.792081 29981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:46.792596 29981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778e64b6233f4a89bdcb6339525c5f89, State: Initialized, Role: FOLLOWER
I20260925 00:55:46.793156 29981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:46.793285 29981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:55:46.793354 29981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:55:46.793460 29981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:55:46.795464 29981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:46.795810 29981 leader_election.cc:302] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778e64b6233f4a89bdcb6339525c5f89; no voters:
I20260925 00:55:46.796110 29981 leader_election.cc:288] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:55:46.796278 29985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:55:46.796531 29985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [term 2 LEADER]: Becoming Leader. State: Replica: 778e64b6233f4a89bdcb6339525c5f89, State: Running, Role: LEADER
I20260925 00:55:46.796916 29985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } }
I20260925 00:55:46.797384 29981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:46.798491 29987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778e64b6233f4a89bdcb6339525c5f89. Latest consensus state: current_term: 2 leader_uuid: "778e64b6233f4a89bdcb6339525c5f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } } }
I20260925 00:55:46.798487 29986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "778e64b6233f4a89bdcb6339525c5f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778e64b6233f4a89bdcb6339525c5f89" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36559 } } }
I20260925 00:55:46.798631 29986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:46.798631 29987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:46.799144 29994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:46.799579 29981 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 00:55:46.805267 29994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:46.806099 29994 catalog_manager.cc:1259] Loaded cluster ID: 744f890061474f5c8b6cce8086edeb80
I20260925 00:55:46.806208 29994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:55:46.808835 29994 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:55:46.809628 29994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Loaded TSK: 0
I20260925 00:55:46.810966 29994 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:55:46.915535 29983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:46.915882 29983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:46.915943 29983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:46.922060 29983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:46.922156 29983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:46.922240 29983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:55:46.930428 29983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=41243
--tserver_master_addrs=127.16.202.62:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:46.931905 29983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:46.934355 29983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:46.948022 30019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:46.948428 30018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:46.948571 29983 server_base.cc:1034] running on GCE node
W20260925 00:55:46.948886 30021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:46.949436 29983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:46.950295 29983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:46.951498 29983 hybrid_clock.cc:648] HybridClock initialized: now 1790297746951462 us; error 52 us; skew 500 ppm
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:46.955044 29983 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:55:46.956261 29983 webserver.cc:460] Webserver started at http://127.16.202.1:41243/ using document root <none> and password file <none>
I20260925 00:55:46.957034 29983 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:46.957144 29983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:55:46 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:47.004084 29983 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.003s sys 0.000s
I20260925 00:55:47.007525 30029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.009333 29983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 00:55:47.009492 29983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "975ed1b8c7de467ab2b40ff2d085d103"
format_stamp: "Formatted at 2026-09-25 00:55:45 on dist-test-slave-rpk1"
server_key: "f6f869e101ccdda49585e9a3776a87aa"
server_key_iv: "c605d4e41141dd306605ebcbca29c85a"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:47.010308 29983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:47.042404 29983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:47.046241 29983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:47.046456 29983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:47.047206 29983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:47.048251 29983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:47.048313 29983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.048367 29983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:47.048393 29983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.059631 29983 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:35199
I20260925 00:55:47.060853 30142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:35199 every 8 connection(s)
I20260925 00:55:47.060911 29983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:55:47.066046 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 29983
I20260925 00:55:47.066587 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:35807
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:36559
--webserver_port=37855
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:47.082973 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.063738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36559 (local address 127.16.202.1:38099)
0925 00:55:47.064445 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:47.064520 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:47.065689 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:47.068155 (+ 2466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:47.068164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:47.068536 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:47.069241 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:47.069265 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.071204 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.071208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:47.072820 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:47.072835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.074223 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.074228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:47.074426 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.075092 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:47.075113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:47.076721 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:47.079295 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:47.079303 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:47.079319 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:47.079658 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:47.080060 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:47.080063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:47.080065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:47.080237 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:47.081126 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:47.081133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:47.081423 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 00:55:47.081610 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 00:55:47.081946 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":151,"threads_started":1}
I20260925 00:55:47.084728 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.063552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:38099 (local address 127.16.202.62:36559)
0925 00:55:47.066286 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:47.066311 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:47.066360 (+ 49us) server_negotiation.cc:407] Connection header received
0925 00:55:47.066700 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:47.066717 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:47.067196 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:47.067611 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:47.069452 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.071023 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.072992 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.073792 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.073964 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.077006 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:47.077013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:47.077036 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:47.077072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:47.079146 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.079793 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.079800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.079805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.079947 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.080344 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.080348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.080350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.080858 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:47.080988 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:47.083523 (+ 2535us) server_negotiation.cc:299] Negotiation successful
0925 00:55:47.083814 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2541,"thread_start_us":127,"threads_started":1}
I20260925 00:55:47.087460 30143 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36559
I20260925 00:55:47.087764 30143 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.088330 30143 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
W20260925 00:55:47.090612 30143 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:36559 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 00:55:47.090742 30143 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.090927 30143 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:47.091863 30143 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.092034 30143 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
W20260925 00:55:47.092775 30143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:55:47.229271 30147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:47.229681 30147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:47.229761 30147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:47.235821 30147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:47.235914 30147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:47.236019 30147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:55:47.244197 30147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=37855
--tserver_master_addrs=127.16.202.62:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:47.245690 30147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:47.247107 30147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:47.260205 30155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:47.260205 30152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:47.260388 30153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:47.261832 30147 server_base.cc:1034] running on GCE node
I20260925 00:55:47.262364 30147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:47.263219 30147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:47.264420 30147 hybrid_clock.cc:648] HybridClock initialized: now 1790297747264403 us; error 33 us; skew 500 ppm
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:47.268399 30147 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:55:47.269706 30147 webserver.cc:460] Webserver started at http://127.16.202.2:37855/ using document root <none> and password file <none>
I20260925 00:55:47.270493 30147 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:47.270635 30147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:47.294155 30147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:55:47.305697 30164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.307829 30147 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.000s
I20260925 00:55:47.308010 30147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "141c38713e384e40b510e56849a85f5e"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "a76024ddef66d173f01d19a819b0198a"
server_key_iv: "5e8431387d94561dcc7a81a02cd09aee"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:47.308856 30147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:47.341346 30147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:47.344636 30147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:47.344928 30147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:47.345564 30147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:47.346441 30147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:47.346486 30147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.346531 30147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:47.346560 30147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.355432 30147 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:35807
I20260925 00:55:47.355473 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:35807 every 8 connection(s)
I20260925 00:55:47.356770 30147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:55:47.357546 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30147
I20260925 00:55:47.358038 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.202.20:43951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:34465
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:36559
--webserver_port=46115
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:47.377828 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.363158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:49737 (local address 127.16.202.62:36559)
0925 00:55:47.363290 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:47.363294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:47.363311 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:55:47.363940 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:47.363944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:47.364004 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:47.364060 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:47.365824 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.366599 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.368271 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.368646 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.368795 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.373020 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:47.373025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:47.373028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:47.373063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:47.375105 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.375866 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.375871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.375873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.375979 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.376405 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.376408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.376410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.376656 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:47.376786 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:47.377503 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 00:55:47.377650 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:55:47.379213 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.361929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36559 (local address 127.16.202.2:49737)
0925 00:55:47.362538 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:47.362598 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:47.363660 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:47.364506 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:47.364515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:47.364929 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:47.365637 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:47.365659 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.366754 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.366758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:47.368143 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:47.368155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.368840 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.368844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:47.369073 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.370783 (+ 1710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:47.370808 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:47.372730 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:47.375277 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:47.375286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:47.375309 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:47.375738 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:47.376084 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:47.376088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:47.376090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:47.376304 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:47.376802 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:47.376809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:47.377210 (+ 401us) client_negotiation.cc:769] Sending connection context
0925 00:55:47.377463 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 00:55:47.378062 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":164,"threads_started":1}
I20260925 00:55:47.380723 30278 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36559
I20260925 00:55:47.380992 30278 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.381577 30278 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
W20260925 00:55:47.383355 30278 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:36559 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 00:55:47.383489 30278 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.383664 30278 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:47.384558 30278 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.384745 30278 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
W20260925 00:55:47.385562 30278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:55:47.500428 30280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:47.500705 30280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:47.500775 30280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:47.505709 30280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:47.505796 30280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:47.505885 30280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:55:47.512605 30280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.202.20:43951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:34465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=46115
--tserver_master_addrs=127.16.202.62:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:47.513741 30280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:47.515074 30280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:47.527915 30287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:47.527930 30286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:47.528882 30280 server_base.cc:1034] running on GCE node
W20260925 00:55:47.529245 30289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:47.529678 30280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:47.530418 30280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:47.531607 30280 hybrid_clock.cc:648] HybridClock initialized: now 1790297747531577 us; error 38 us; skew 500 ppm
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:47.535761 30280 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:55:47.537106 30280 webserver.cc:460] Webserver started at http://127.16.202.3:46115/ using document root <none> and password file <none>
I20260925 00:55:47.537938 30280 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:47.538097 30280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:47.557914 30280 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:55:47.562978 30297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.564797 30280 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260925 00:55:47.565033 30280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "b8b09cae577143948e6873593b411306"
format_stamp: "Formatted at 2026-09-25 00:55:46 on dist-test-slave-rpk1"
server_key: "b0b807c47d1e36167acd73caa4677d61"
server_key_iv: "eaf4543d88a1bd1978d2f2eab16efeb2"
server_key_version: "kuduclusterkey@0"
I20260925 00:55:47.565810 30280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:47.584409 30280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:47.588110 30280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:47.588325 30280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:47.589064 30280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:47.589946 30280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:47.589994 30280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.590046 30280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:47.590075 30280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:47.601193 30280 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:34465
I20260925 00:55:47.601233 30417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:34465 every 8 connection(s)
I20260925 00:55:47.602314 30280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:55:47.603742 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30280
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:47.614651 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.604459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.16.202.62:36559)
0925 00:55:47.604773 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:47.604783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:47.605081 (+ 298us) server_negotiation.cc:407] Connection header received
0925 00:55:47.605321 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:47.605325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:47.605379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:47.605440 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:47.606486 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.607323 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.608507 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.608862 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.608956 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.610634 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:47.610638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:47.610641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:47.610670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:47.612793 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.613319 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.613323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.613324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.613426 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.613740 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.613744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.613746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.613948 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:47.614145 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:47.614325 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 00:55:47.614465 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260925 00:55:47.620306 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.604194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:47305 (local address 127.16.202.62:36559)
0925 00:55:47.604329 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:47.604345 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:47.606179 (+ 1834us) server_negotiation.cc:407] Connection header received
0925 00:55:47.607311 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:47.607314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:47.607362 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:47.607508 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:47.609201 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.609982 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.611694 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.611925 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.612013 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.616003 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:47.616007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:47.616010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:47.616036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:47.617727 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.618394 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.618398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.618400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.618480 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:47.618913 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:47.618917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:47.618919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:47.619134 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:47.619215 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:47.619954 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 00:55:47.620094 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:55:47.621320 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:47.604792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36559 (local address 127.16.202.3:47305)
0925 00:55:47.605998 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:47.606059 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:47.607080 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:47.607935 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:47.607945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:47.608285 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:47.609031 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:47.609054 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.610145 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.610149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:47.611578 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:47.611590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:47.612080 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:47.612084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:47.612302 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:47.612933 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:47.612955 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:47.615549 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:47.617866 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:47.617885 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:47.617905 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:47.618276 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:47.618588 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:47.618592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:47.618594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:47.618798 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:47.619281 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:47.619288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:47.619641 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 00:55:47.619850 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 00:55:47.620213 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":127,"threads_started":1}
I20260925 00:55:47.622555 30418 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36559
I20260925 00:55:47.622833 30418 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:47.623426 30418 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:47.624907 29945 ts_manager.cc:194] Registered new tserver with Master: b8b09cae577143948e6873593b411306 (127.16.202.3:34465)
I20260925 00:55:47.626519 29945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:47305
Sep 25 00:55:47 dist-test-slave-rpk1 krb5kdc[28949](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:48.093000 30143 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:48.093305 30143 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:48.094023 29945 ts_manager.cc:194] Registered new tserver with Master: 975ed1b8c7de467ab2b40ff2d085d103 (127.16.202.1:35199)
I20260925 00:55:48.094827 29945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:38099
Sep 25 00:55:48 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:48.385833 30278 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:48.386179 30278 heartbeater.cc:511] Master 127.16.202.62:36559 requested a full tablet report, sending...
I20260925 00:55:48.387102 29945 ts_manager.cc:194] Registered new tserver with Master: 141c38713e384e40b510e56849a85f5e (127.16.202.2:35807)
I20260925 00:55:48.387698 29945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:49737
I20260925 00:55:48.391666 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:55:48.404506 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.393357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.16.202.62:36559)
0925 00:55:48.394983 (+ 1626us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.394988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.395001 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:55:48.395041 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.395044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.395090 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.395175 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:48.396179 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.397046 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.398304 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.398733 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.398827 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.400573 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:48.400576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:48.400579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:48.400608 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:48.402453 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:48.402975 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:48.402978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:48.402980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:48.403066 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:48.403343 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:48.403346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:48.403348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:48.403556 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:48.403678 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.404192 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.404322 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":89,"threads_started":1}
I20260925 00:55:48.408914 29945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:55:48.413384 29945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:55:48.435117 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.425850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:35807 (local address 127.0.0.1:57876)
0925 00:55:48.428917 (+ 3067us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:48.428930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:48.429091 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:48.429677 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:48.429680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:48.429697 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:48.429849 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.429856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.431288 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.431291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.433314 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.433325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.434553 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.434556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.434718 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.434739 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:55:48.434831 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:55:48.434957 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"mutex_wait_us":29,"thread_start_us":100,"threads_started":1}
I20260925 00:55:48.435168 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.427230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.16.202.1:35199)
0925 00:55:48.427632 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.427639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.428541 (+ 902us) server_negotiation.cc:407] Connection header received
0925 00:55:48.428908 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.428915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.429205 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.429386 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:48.429941 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.431368 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.433346 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.434189 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.434270 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.434435 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.434913 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.435019 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":94,"threads_started":1}
I20260925 00:55:48.436000 30440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.427064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:35199 (local address 127.0.0.1:40152)
0925 00:55:48.428364 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:48.428440 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:48.428665 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:48.429634 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:48.429639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:48.429664 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:48.429805 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.429814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.431520 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.431523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.432925 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.432936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.434538 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.434542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.434687 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.434730 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 00:55:48.435785 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0925 00:55:48.435867 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260925 00:55:48.436448 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.425427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.16.202.2:35807)
0925 00:55:48.427849 (+ 2422us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.427855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.429021 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 00:55:48.429222 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.429227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.429404 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.429513 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:48.429976 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.431152 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.433448 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.434258 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.434333 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.434461 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.436220 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.436318 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2298,"thread_start_us":144,"threads_started":1}
I20260925 00:55:48.438921 30208 tablet_service.cc:1467] Processing CreateTablet for tablet 20c5634f351049419857efcc889ad3da (DEFAULT_TABLE table=test-table [id=6c98c17ee3da4b62ba536dc682f45590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:48.439109 30077 tablet_service.cc:1467] Processing CreateTablet for tablet 20c5634f351049419857efcc889ad3da (DEFAULT_TABLE table=test-table [id=6c98c17ee3da4b62ba536dc682f45590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:48.439957 30208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c5634f351049419857efcc889ad3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:48.440116 30077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c5634f351049419857efcc889ad3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:48.442276 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.427793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:34465 (local address 127.0.0.1:49706)
0925 00:55:48.433149 (+ 5356us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:48.433163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:48.433240 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:48.438163 (+ 4923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:48.438166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:48.438184 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:48.438270 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.438277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.439633 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.439637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.440781 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.440793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.441864 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.441868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.441976 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.442005 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:55:48.442056 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 00:55:48.442130 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5291,"thread_start_us":92,"threads_started":1}
I20260925 00:55:48.443022 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.427956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.16.202.3:34465)
0925 00:55:48.437561 (+ 9605us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.437567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.437592 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:55:48.437665 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.437670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.437860 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.438017 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:48.438407 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.439503 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.440937 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.441725 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.442574 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.442731 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.442781 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.442886 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9493,"thread_start_us":69,"threads_started":1}
I20260925 00:55:48.446120 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 20c5634f351049419857efcc889ad3da (DEFAULT_TABLE table=test-table [id=6c98c17ee3da4b62ba536dc682f45590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:48.447309 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c5634f351049419857efcc889ad3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:48.450995 30444 tablet_bootstrap.cc:492] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103: Bootstrap starting.
I20260925 00:55:48.453478 30444 tablet_bootstrap.cc:654] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:48.454384 30444 log.cc:824] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:48.455368 30446 tablet_bootstrap.cc:492] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Bootstrap starting.
I20260925 00:55:48.457672 30446 tablet_bootstrap.cc:654] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:48.458521 30446 log.cc:824] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:48.460932 30446 tablet_bootstrap.cc:492] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: No bootstrap required, opened a new log
I20260925 00:55:48.461159 30446 ts_tablet_manager.cc:1397] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:55:48.464233 30444 tablet_bootstrap.cc:492] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103: No bootstrap required, opened a new log
I20260925 00:55:48.464440 30444 ts_tablet_manager.cc:1397] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260925 00:55:48.465500 30446 raft_consensus.cc:348] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.465792 30446 raft_consensus.cc:374] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:48.465847 30446 raft_consensus.cc:729] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b09cae577143948e6873593b411306, State: Initialized, Role: FOLLOWER
I20260925 00:55:48.466373 30446 consensus_queue.cc:260] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.467523 30446 ts_tablet_manager.cc:1428] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:55:48.468027 30418 heartbeater.cc:503] Master 127.16.202.62:36559 was elected leader, sending a full tablet report...
I20260925 00:55:48.469050 30444 raft_consensus.cc:348] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.469394 30444 raft_consensus.cc:374] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:48.469480 30444 raft_consensus.cc:729] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ed1b8c7de467ab2b40ff2d085d103, State: Initialized, Role: FOLLOWER
I20260925 00:55:48.469991 30444 consensus_queue.cc:260] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.471115 30444 ts_tablet_manager.cc:1428] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:55:48.471494 30143 heartbeater.cc:503] Master 127.16.202.62:36559 was elected leader, sending a full tablet report...
I20260925 00:55:48.475193 30448 tablet_bootstrap.cc:492] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Bootstrap starting.
I20260925 00:55:48.478035 30448 tablet_bootstrap.cc:654] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:48.478989 30448 log.cc:824] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:48.482079 30448 tablet_bootstrap.cc:492] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: No bootstrap required, opened a new log
I20260925 00:55:48.482331 30448 ts_tablet_manager.cc:1397] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:55:48.486616 30448 raft_consensus.cc:348] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.486953 30448 raft_consensus.cc:374] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:48.487000 30448 raft_consensus.cc:729] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141c38713e384e40b510e56849a85f5e, State: Initialized, Role: FOLLOWER
I20260925 00:55:48.487563 30448 consensus_queue.cc:260] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.488394 30278 heartbeater.cc:503] Master 127.16.202.62:36559 was elected leader, sending a full tablet report...
I20260925 00:55:48.488785 30448 ts_tablet_manager.cc:1428] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 00:55:48.490962 30450 raft_consensus.cc:482] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:55:48.491168 30450 raft_consensus.cc:504] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.492473 30450 leader_election.cc:288] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 141c38713e384e40b510e56849a85f5e (127.16.202.2:35807), b8b09cae577143948e6873593b411306 (127.16.202.3:34465)
I20260925 00:55:48.497655 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.492981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:42501 (local address 127.16.202.2:35807)
0925 00:55:48.493096 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.493100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.493321 (+ 221us) server_negotiation.cc:407] Connection header received
0925 00:55:48.493490 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.493493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.493533 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.493572 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:48.493957 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.494698 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.496238 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.496821 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.496906 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.496966 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.497463 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.497538 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:55:48.498225 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.493702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:39753 (local address 127.16.202.3:34465)
0925 00:55:48.493806 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.493810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.494011 (+ 201us) server_negotiation.cc:407] Connection header received
0925 00:55:48.494204 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.494207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.494253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.494299 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:48.494690 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.495400 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.496883 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.497499 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.497594 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.497643 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.498025 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.498106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 00:55:48.498489 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.492921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:35807 (local address 127.16.202.1:42501)
0925 00:55:48.493250 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:48.493263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:48.493386 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:48.493696 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:48.493699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:48.493715 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:48.493844 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.493849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.494826 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.494829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.496106 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.496117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.497229 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.497232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.497328 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.497352 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:55:48.498076 (+ 724us) client_negotiation.cc:240] Negotiation successful
0925 00:55:48.498156 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":7,"thread_start_us":156,"threads_started":1}
I20260925 00:55:48.498667 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.493348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:34465 (local address 127.16.202.1:39753)
0925 00:55:48.493924 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:48.493938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:48.494076 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:48.494434 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:48.494437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:48.494453 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:48.494570 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.494576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.495572 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.495575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.496732 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:48.496743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.497735 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.497738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:48.497840 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.497903 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 00:55:48.498413 (+ 510us) client_negotiation.cc:240] Negotiation successful
0925 00:55:48.498489 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":84,"threads_started":1}
I20260925 00:55:48.499497 30371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c5634f351049419857efcc889ad3da" candidate_uuid: "975ed1b8c7de467ab2b40ff2d085d103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b09cae577143948e6873593b411306" is_pre_election: true
I20260925 00:55:48.499496 30230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c5634f351049419857efcc889ad3da" candidate_uuid: "975ed1b8c7de467ab2b40ff2d085d103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141c38713e384e40b510e56849a85f5e" is_pre_election: true
I20260925 00:55:48.499835 30371 raft_consensus.cc:2454] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ed1b8c7de467ab2b40ff2d085d103 in term 0.
I20260925 00:55:48.499836 30230 raft_consensus.cc:2454] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ed1b8c7de467ab2b40ff2d085d103 in term 0.
I20260925 00:55:48.500430 30031 leader_election.cc:302] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975ed1b8c7de467ab2b40ff2d085d103, b8b09cae577143948e6873593b411306; no voters:
I20260925 00:55:48.500803 30450 raft_consensus.cc:2781] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:55:48.500934 30450 raft_consensus.cc:482] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:55:48.501001 30450 raft_consensus.cc:3037] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:48.502394 30450 raft_consensus.cc:504] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.502784 30450 leader_election.cc:288] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [CANDIDATE]: Term 1 election: Requested vote from peers 141c38713e384e40b510e56849a85f5e (127.16.202.2:35807), b8b09cae577143948e6873593b411306 (127.16.202.3:34465)
I20260925 00:55:48.503525 30230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c5634f351049419857efcc889ad3da" candidate_uuid: "975ed1b8c7de467ab2b40ff2d085d103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141c38713e384e40b510e56849a85f5e"
I20260925 00:55:48.503721 30230 raft_consensus.cc:3037] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:48.505148 30230 raft_consensus.cc:2454] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ed1b8c7de467ab2b40ff2d085d103 in term 1.
I20260925 00:55:48.505483 30031 leader_election.cc:302] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141c38713e384e40b510e56849a85f5e, 975ed1b8c7de467ab2b40ff2d085d103; no voters:
I20260925 00:55:48.505537 30371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c5634f351049419857efcc889ad3da" candidate_uuid: "975ed1b8c7de467ab2b40ff2d085d103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b09cae577143948e6873593b411306"
I20260925 00:55:48.505686 30371 raft_consensus.cc:3037] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:48.505755 30450 raft_consensus.cc:2781] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:48.505997 30450 raft_consensus.cc:686] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [term 1 LEADER]: Becoming Leader. State: Replica: 975ed1b8c7de467ab2b40ff2d085d103, State: Running, Role: LEADER
I20260925 00:55:48.506384 30450 consensus_queue.cc:237] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } }
I20260925 00:55:48.507149 30371 raft_consensus.cc:2454] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ed1b8c7de467ab2b40ff2d085d103 in term 1.
I20260925 00:55:48.509967 29944 catalog_manager.cc:5495] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ed1b8c7de467ab2b40ff2d085d103 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "975ed1b8c7de467ab2b40ff2d085d103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ed1b8c7de467ab2b40ff2d085d103" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 35199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 } health_report { overall_health: UNKNOWN } } }
I20260925 00:55:48.533748 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.529011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.16.202.1:35199)
0925 00:55:48.529159 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:48.529163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:48.529179 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:55:48.529283 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:48.529287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:48.529380 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:48.529508 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:55:48.529888 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.530651 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.531878 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:48.532195 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:48.532291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:48.532699 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:55:48.533277 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:55:48.533397 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:48.533527 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 00:55:48.533592 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 00:55:48.541275 30230 raft_consensus.cc:1261] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 1 FOLLOWER]: Refusing update from remote peer 975ed1b8c7de467ab2b40ff2d085d103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:48.542055 30450 consensus_queue.cc:1035] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141c38713e384e40b510e56849a85f5e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:48.542296 30371 raft_consensus.cc:1261] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 1 FOLLOWER]: Refusing update from remote peer 975ed1b8c7de467ab2b40ff2d085d103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:48.542939 30455 consensus_queue.cc:1035] T 20c5634f351049419857efcc889ad3da P 975ed1b8c7de467ab2b40ff2d085d103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8b09cae577143948e6873593b411306" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:48.555452 30458 mvcc.cc:205] Tried to move back new op lower bound from 7333059578016534528 to 7333059577886179328. Current Snapshot: MvccSnapshot[applied={T|T < 7333059578016534528}]
I20260925 00:55:48.558729 30459 mvcc.cc:205] Tried to move back new op lower bound from 7333059578016534528 to 7333059577886179328. Current Snapshot: MvccSnapshot[applied={T|T < 7333059578016534528}]
I20260925 00:55:48.567251 30460 mvcc.cc:205] Tried to move back new op lower bound from 7333059578016534528 to 7333059577886179328. Current Snapshot: MvccSnapshot[applied={T|T < 7333059578016534528}]
I20260925 00:55:48.569124 29944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:48.569320 29944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:48.572540 29944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778e64b6233f4a89bdcb6339525c5f89: Sending DeleteTablet for 3 replicas of tablet 20c5634f351049419857efcc889ad3da
I20260925 00:55:48.573500 30208 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c5634f351049419857efcc889ad3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:48 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:57876
I20260925 00:55:48.573745 30077 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c5634f351049419857efcc889ad3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:48 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:40152
I20260925 00:55:48.575162 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 29983
I20260925 00:55:48.575145 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c5634f351049419857efcc889ad3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:48 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:49706
I20260925 00:55:48.576889 30471 tablet_replica.cc:330] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: stopping tablet replica
I20260925 00:55:48.577282 30471 raft_consensus.cc:2229] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:48.577607 30471 raft_consensus.cc:2258] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:48.578073 30470 tablet_replica.cc:330] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: stopping tablet replica
I20260925 00:55:48.578418 30470 raft_consensus.cc:2229] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:48.578747 30470 raft_consensus.cc:2258] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:48.580510 30470 ts_tablet_manager.cc:1905] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:55:48.583391 30470 ts_tablet_manager.cc:1918] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:48.583494 30470 log.cc:1196] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/20c5634f351049419857efcc889ad3da
I20260925 00:55:48.583799 30470 ts_tablet_manager.cc:1939] T 20c5634f351049419857efcc889ad3da P 141c38713e384e40b510e56849a85f5e: Deleting consensus metadata
I20260925 00:55:48.585209 29930 catalog_manager.cc:4841] TS 141c38713e384e40b510e56849a85f5e (127.16.202.2:35807): tablet 20c5634f351049419857efcc889ad3da (table test-table [id=6c98c17ee3da4b62ba536dc682f45590]) successfully deleted
I20260925 00:55:48.587533 30471 ts_tablet_manager.cc:1905] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:55:48.589075 29932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:35199 (error 108)
I20260925 00:55:48.589908 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30147
I20260925 00:55:48.590731 30471 ts_tablet_manager.cc:1918] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:48.590840 30471 log.cc:1196] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/20c5634f351049419857efcc889ad3da
I20260925 00:55:48.591162 30471 ts_tablet_manager.cc:1939] T 20c5634f351049419857efcc889ad3da P b8b09cae577143948e6873593b411306: Deleting consensus metadata
I20260925 00:55:48.592614 29930 catalog_manager.cc:4841] TS b8b09cae577143948e6873593b411306 (127.16.202.3:34465): tablet 20c5634f351049419857efcc889ad3da (table test-table [id=6c98c17ee3da4b62ba536dc682f45590]) successfully deleted
I20260925 00:55:48.593721 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:48.593376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:35199 (local address 127.0.0.1:40178)
0925 00:55:48.593528 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:48.593622 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 00:55:48.594146 29932 catalog_manager.cc:4584] TS 975ed1b8c7de467ab2b40ff2d085d103 (127.16.202.1:35199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20c5634f351049419857efcc889ad3da: Network error: Client connection negotiation failed: client connection to 127.16.202.1:35199: connect: Connection refused (error 111)
Sep 25 00:55:48 dist-test-slave-rpk1 krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for HTTP/127.16.202.20@KRBTEST.COM
I20260925 00:55:48.599004 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30280
I20260925 00:55:48.605504 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 29912
I20260925 00:55:48.613276 17192 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 00:55:48.960938 17192 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 00:55:48.961098 17192 mini_ranger.cc:66] Stopping Ranger...
I20260925 00:55:49.296643 17192 mini_ranger.cc:68] Stopped Ranger
2026-09-25 00:55:49.296 UTC [28979] LOG: received smart shutdown request
2026-09-25 00:55:49.300 UTC [28979] LOG: background worker "logical replication launcher" (PID 28988) exited with exit code 1
2026-09-25 00:55:49.301 UTC [28983] LOG: shutting down
2026-09-25 00:55:49.478 UTC [28979] LOG: database system is shut down
2026-09-25T00:55:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48528 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:55:49 dist-test-slave-rpk1 krb5kdc[30482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:55:49 dist-test-slave-rpk1 krb5kdc[30482](info): set up 2 sockets
Sep 25 00:55:49 dist-test-slave-rpk1 krb5kdc[30482](info): commencing operation
krb5kdc: starting...
W20260925 00:55:50.532598 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:55:50 dist-test-slave-rpk1 krb5kdc[30482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:55:50.656206 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39313
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39313,127.16.202.61:34139,127.16.202.60:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:50.767210 30495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:50.767521 30495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:50.767588 30495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:50.771785 30495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:50.771855 30495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:50.771883 30495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:50.771906 30495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:50.771936 30495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:50.777401 30495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39313,127.16.202.61:34139,127.16.202.60:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:50.778494 30495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:50.779536 30495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:50.788120 30503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:50.788331 30495 server_base.cc:1034] running on GCE node
W20260925 00:55:50.788120 30501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:50.788121 30500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:50.789089 30495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:50.790232 30495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:50.791421 30495 hybrid_clock.cc:648] HybridClock initialized: now 1790297750791392 us; error 44 us; skew 500 ppm
Sep 25 00:55:50 dist-test-slave-rpk1 krb5kdc[30482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:50.794328 30495 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:50.795272 30495 webserver.cc:460] Webserver started at http://127.16.202.62:41015/ using document root <none> and password file <none>
I20260925 00:55:50.795845 30495 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:50.795912 30495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:50.796128 30495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:50.798233 30495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7"
format_stamp: "Formatted at 2026-09-25 00:55:50 on dist-test-slave-rpk1"
server_key: "2f3f3850d5d3db2d96c4419c5ae68c67"
server_key_iv: "0ddc25e82af09770ac149d4554c8f784"
server_key_version: "encryptionkey@0"
I20260925 00:55:50.798820 30495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7"
format_stamp: "Formatted at 2026-09-25 00:55:50 on dist-test-slave-rpk1"
server_key: "2f3f3850d5d3db2d96c4419c5ae68c67"
server_key_iv: "0ddc25e82af09770ac149d4554c8f784"
server_key_version: "encryptionkey@0"
I20260925 00:55:50.802609 30495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:55:50.805027 30510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:50.806288 30495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 00:55:50.806407 30495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7"
format_stamp: "Formatted at 2026-09-25 00:55:50 on dist-test-slave-rpk1"
server_key: "2f3f3850d5d3db2d96c4419c5ae68c67"
server_key_iv: "0ddc25e82af09770ac149d4554c8f784"
server_key_version: "encryptionkey@0"
I20260925 00:55:50.806510 30495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:50.824920 30495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:50.828193 30495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:50.828389 30495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:50.835662 30495 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39313
I20260925 00:55:50.835773 30562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39313 every 8 connection(s)
I20260925 00:55:50.836742 30495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:50.839671 30563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:50.842908 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30495
I20260925 00:55:50.843009 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:55:50.843257 17192 external_mini_cluster.cc:1428] Setting key 0515127afff9f107bcee6bb670cca64d
I20260925 00:55:50.843581 30563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.202.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.61@KRBTEST.COM" created.
Sep 25 00:55:50 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:50.861006 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.846045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39313 (local address 127.0.0.1:33698)
0925 00:55:50.846624 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.846676 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:50.847738 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:50.848614 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:50.848621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:50.849054 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:50.849319 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:50.849333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:50.850995 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:50.850998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:50.851794 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:50.851805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:50.852754 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:50.852763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:50.852863 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:50.853494 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:50.853512 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:50.856046 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:50.858063 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:50.858072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:50.858084 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:50.858369 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:50.858673 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:50.858676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:50.858678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:50.858816 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:50.859490 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:50.859497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:50.859664 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 00:55:50.859862 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 00:55:50.860264 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":70,"thread_start_us":121,"threads_started":1}
I20260925 00:55:50.861003 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.846045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.16.202.62:39313)
0925 00:55:50.846521 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:50.846538 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:50.846791 (+ 253us) server_negotiation.cc:407] Connection header received
0925 00:55:50.848104 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:50.848109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:50.848348 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:50.848554 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:50.849478 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:50.850847 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:50.851927 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:50.852531 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:50.852697 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:50.856221 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:50.856228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:50.856245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:50.856272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:50.857941 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:50.858486 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:50.858491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:50.858492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:50.858597 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:50.858896 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:50.858898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:50.858899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:50.859366 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:50.859475 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:50.859929 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 00:55:50.860145 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"spinlock_wait_cycles":20736,"thread_start_us":168,"threads_started":1}
I20260925 00:55:50.863502 30563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:55:50.864200 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.863920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39014)
0925 00:55:50.864036 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.864105 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 00:55:50.864368 30513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
I20260925 00:55:50.865229 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.864993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39018)
0925 00:55:50.865132 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.865178 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 00:55:50.865352 30563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:34139: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab.
Entry for principal kudu/127.16.202.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab.
I20260925 00:55:50.872165 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:34139
--webserver_interface=127.16.202.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab
--principal=kudu/127.16.202.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39313,127.16.202.61:34139,127.16.202.60:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:50.914624 30563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } attempt: 1
I20260925 00:55:50.915562 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.915214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39034)
0925 00:55:50.915409 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.915477 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 00:55:50.916296 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.916090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39040)
0925 00:55:50.916193 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.916239 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 00:55:50.916414 30563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:34139: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
I20260925 00:55:50.984589 30563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } attempt: 2
W20260925 00:55:50.984983 30570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:50.985394 30570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260925 00:55:50.985428 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.985129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39042)
0925 00:55:50.985275 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.985335 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 00:55:50.985469 30570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 00:55:50.986120 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:50.985900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39050)
0925 00:55:50.986009 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:50.986059 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:55:50.986289 30563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:34139: Network error: Client connection negotiation failed: client connection to 127.16.202.61:34139: connect: Connection refused (error 111)
W20260925 00:55:50.989836 30570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:50.989899 30570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:50.989918 30570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:50.989933 30570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:50.989948 30570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:50.995533 30570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39313,127.16.202.61:34139,127.16.202.60:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab
--principal=kudu/127.16.202.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.202.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:50.996604 30570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:50.997674 30570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:51.006399 30575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:51.006405 30578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:51.006520 30570 server_base.cc:1034] running on GCE node
W20260925 00:55:51.006460 30576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:51.007122 30570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:51.008293 30570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:51.009514 30570 hybrid_clock.cc:648] HybridClock initialized: now 1790297751009495 us; error 35 us; skew 500 ppm
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:51.012454 30570 init.cc:377] Logged in from keytab as kudu/127.16.202.61@KRBTEST.COM (short username kudu)
I20260925 00:55:51.013422 30570 webserver.cc:460] Webserver started at http://127.16.202.61:36783/ using document root <none> and password file <none>
I20260925 00:55:51.013960 30570 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:51.014027 30570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:51.014225 30570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:51.016348 30570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data/instance:
uuid: "e68a1051028444768518b834338a6fdb"
format_stamp: "Formatted at 2026-09-25 00:55:51 on dist-test-slave-rpk1"
server_key: "d5df1af93c4de596b69a7c7184af9890"
server_key_iv: "7761c0a20a503b7a1b4cf6bcb36d37d5"
server_key_version: "encryptionkey@0"
I20260925 00:55:51.016932 30570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal/instance:
uuid: "e68a1051028444768518b834338a6fdb"
format_stamp: "Formatted at 2026-09-25 00:55:51 on dist-test-slave-rpk1"
server_key: "d5df1af93c4de596b69a7c7184af9890"
server_key_iv: "7761c0a20a503b7a1b4cf6bcb36d37d5"
server_key_version: "encryptionkey@0"
I20260925 00:55:51.020710 30570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:55:51.023437 30585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:51.024811 30570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:51.024940 30570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal
uuid: "e68a1051028444768518b834338a6fdb"
format_stamp: "Formatted at 2026-09-25 00:55:51 on dist-test-slave-rpk1"
server_key: "d5df1af93c4de596b69a7c7184af9890"
server_key_iv: "7761c0a20a503b7a1b4cf6bcb36d37d5"
server_key_version: "encryptionkey@0"
I20260925 00:55:51.025048 30570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:51.042197 30570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:51.045496 30570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:51.045686 30570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:51.052906 30570 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.61:34139
I20260925 00:55:51.052929 30637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.61:34139 every 8 connection(s)
I20260925 00:55:51.053893 30570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
I20260925 00:55:51.056756 30638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:51.058656 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30570
I20260925 00:55:51.058772 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-1/wal/instance
I20260925 00:55:51.059044 17192 external_mini_cluster.cc:1428] Setting key fff530d31667cfbc9cb0565bae85b2ba
I20260925 00:55:51.061448 30638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.202.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.60@KRBTEST.COM" created.
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:51.077456 30563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } attempt: 3
I20260925 00:55:51.077708 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.063633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.16.202.62:39313)
0925 00:55:51.063808 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.063813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.064783 (+ 970us) server_negotiation.cc:407] Connection header received
0925 00:55:51.065808 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.065813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.065878 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.065986 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.067298 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.067842 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.069366 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.069665 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.069785 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.072742 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.072746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.072748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.072779 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.075188 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.075855 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.075859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.075861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.075961 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.076372 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.076375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.076377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.076626 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.076746 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.077374 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.077541 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:55:51.078758 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.064050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39313 (local address 127.0.0.1:33700)
0925 00:55:51.064578 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.064634 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.065573 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.066159 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.066167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.066588 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.067138 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.067153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.068029 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.068033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.069239 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.069250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.069885 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.069891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.070087 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.070781 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.070802 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.072492 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.075349 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.075355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.075439 (+ 84us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.075744 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.076082 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.076085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.076087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.076270 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.076778 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.076784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.077101 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.077278 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.077662 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260925 00:55:51.080291 30638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab.
Entry for principal kudu/127.16.202.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab.
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:55:51.086088 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:37947
--webserver_interface=127.16.202.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab
--principal=kudu/127.16.202.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39313,127.16.202.61:34139,127.16.202.60:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:51.088179 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.077915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39052)
0925 00:55:51.078183 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.078196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.078262 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.078948 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.078951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.079130 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.079259 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.079267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.080590 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.080592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.081338 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.081349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.081932 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.081936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.082040 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.082540 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.082554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.084135 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.086552 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.086556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.086558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.086794 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.087094 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.087097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.087099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.087183 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.087666 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.087669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.087775 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.087868 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.088026 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260925 00:55:51.088904 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.078164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.16.202.61:34139)
0925 00:55:51.078541 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.078551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.078573 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:55:51.078610 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.078614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.078793 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.079383 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.079476 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.080469 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.081432 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.081822 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.081910 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.084594 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.084605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.084627 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.084648 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.086432 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.086889 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.086894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.086896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.086986 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.087265 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.087269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.087271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.087570 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.087687 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.088488 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.088729 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":155,"threads_started":1}
I20260925 00:55:51.090286 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.080795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.16.202.61:34139)
0925 00:55:51.081119 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.081127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.081139 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:55:51.081170 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.081173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.081210 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.081286 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.081731 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.082339 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.083320 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.083757 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.083850 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.086094 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.086098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.086101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.086127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.088754 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.089175 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.089178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.089180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.089250 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.089488 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.089492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.089494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.089698 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.089784 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.090020 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.090130 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":132,"threads_started":1}
I20260925 00:55:51.090742 30563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:55:51.090893 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.080778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39056)
0925 00:55:51.081023 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.081036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.081100 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.081331 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.081334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.081538 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.081627 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.081632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.082435 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.082439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.083206 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.083218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.083950 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.083953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.084061 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.084526 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.084539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.085944 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.088861 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.088864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.088866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.089080 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.089332 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.089334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.089336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.089401 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.089813 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.089815 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.089881 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.090610 (+ 729us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.090741 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 00:55:51.091471 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.091219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42690)
0925 00:55:51.091343 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.091398 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 00:55:51.091873 30638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:55:51.092239 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.091975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42700)
0925 00:55:51.092124 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.092175 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 00:55:51.092446 30563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:37947: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
I20260925 00:55:51.092512 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.092243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42706)
0925 00:55:51.092365 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.092448 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 00:55:51.092739 30586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
I20260925 00:55:51.093603 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.093381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42714)
0925 00:55:51.093479 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.093531 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 00:55:51.093758 30638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:37947: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
I20260925 00:55:51.123631 30563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } attempt: 1
I20260925 00:55:51.124576 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.124213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42718)
0925 00:55:51.124391 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.124468 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 00:55:51.125337 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.125115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42724)
0925 00:55:51.125219 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.125273 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 00:55:51.125504 30563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:37947: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
I20260925 00:55:51.143033 30638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } attempt: 1
I20260925 00:55:51.143934 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.143610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42734)
0925 00:55:51.143788 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.143853 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 00:55:51.144634 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.144423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42738)
0925 00:55:51.144532 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.144575 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:55:51.144763 30638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:37947: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
I20260925 00:55:51.200754 30563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } attempt: 2
I20260925 00:55:51.201612 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.201310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42744)
0925 00:55:51.201456 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.201525 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 00:55:51.202294 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.202082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42752)
0925 00:55:51.202189 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.202235 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:55:51.202456 30563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:37947: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
W20260925 00:55:51.203243 30645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:51.203485 30645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:51.203539 30645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:51.207650 30645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:51.207710 30645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:51.207728 30645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:51.207744 30645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:51.207762 30645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:51.212908 30638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } attempt: 2
I20260925 00:55:51.213259 30645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39313,127.16.202.61:34139,127.16.202.60:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab
--principal=kudu/127.16.202.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.202.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:51.213794 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.213358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42756)
0925 00:55:51.213604 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.213703 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 00:55:51.214582 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.214352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42758)
0925 00:55:51.214454 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.214507 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 00:55:51.214641 30645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260925 00:55:51.214716 30638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:37947: Network error: Client connection negotiation failed: client connection to 127.16.202.60:37947: connect: Connection refused (error 111)
I20260925 00:55:51.215777 30645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:51.224349 30651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:51.224349 30652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:51.224349 30654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:51.224418 30645 server_base.cc:1034] running on GCE node
I20260925 00:55:51.225241 30645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:51.226416 30645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:51.227586 30645 hybrid_clock.cc:648] HybridClock initialized: now 1790297751227559 us; error 47 us; skew 500 ppm
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:51.230504 30645 init.cc:377] Logged in from keytab as kudu/127.16.202.60@KRBTEST.COM (short username kudu)
I20260925 00:55:51.231454 30645 webserver.cc:460] Webserver started at http://127.16.202.60:41885/ using document root <none> and password file <none>
I20260925 00:55:51.232019 30645 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:51.232087 30645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:51.232290 30645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:51.234377 30645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data/instance:
uuid: "768a4154e64940e9a528504da71e6933"
format_stamp: "Formatted at 2026-09-25 00:55:51 on dist-test-slave-rpk1"
server_key: "1bb13fc2ab3b5cba0571f71f9bf20827"
server_key_iv: "f83b4b69966a385ee0f4ae4560c58bd5"
server_key_version: "encryptionkey@0"
I20260925 00:55:51.234901 30645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal/instance:
uuid: "768a4154e64940e9a528504da71e6933"
format_stamp: "Formatted at 2026-09-25 00:55:51 on dist-test-slave-rpk1"
server_key: "1bb13fc2ab3b5cba0571f71f9bf20827"
server_key_iv: "f83b4b69966a385ee0f4ae4560c58bd5"
server_key_version: "encryptionkey@0"
I20260925 00:55:51.238715 30645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 00:55:51.241336 30661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:51.242643 30645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:55:51.242779 30645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal
uuid: "768a4154e64940e9a528504da71e6933"
format_stamp: "Formatted at 2026-09-25 00:55:51 on dist-test-slave-rpk1"
server_key: "1bb13fc2ab3b5cba0571f71f9bf20827"
server_key_iv: "f83b4b69966a385ee0f4ae4560c58bd5"
server_key_version: "encryptionkey@0"
I20260925 00:55:51.242897 30645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:51.258615 30645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:51.261865 30645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:51.262091 30645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:51.268981 30645 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.60:37947
I20260925 00:55:51.269001 30713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.60:37947 every 8 connection(s)
I20260925 00:55:51.269946 30645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
I20260925 00:55:51.272905 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:51.273828 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30645
I20260925 00:55:51.273938 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297591180183-17192-0/minicluster-data/master-2/wal/instance
I20260925 00:55:51.274246 17192 external_mini_cluster.cc:1428] Setting key 319b15e8811176902f5bdd35b1d8220d
I20260925 00:55:51.277403 30714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:51.287773 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.279224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.16.202.62:39313)
0925 00:55:51.279339 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.279342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.279758 (+ 416us) server_negotiation.cc:407] Connection header received
0925 00:55:51.279883 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.279886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.279942 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.280019 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.280708 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.281200 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.282089 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.282300 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.282386 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.284764 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.284768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.284771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.284801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.286036 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.286461 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.286464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.286465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.286536 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.286776 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.286778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.286779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.286966 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.287084 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.287558 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.287655 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:55:51.288631 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.275650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.16.202.60:37947)
0925 00:55:51.276107 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.276122 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.276158 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:55:51.276889 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.276909 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.277267 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.277586 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.278398 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.279764 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.280745 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.281237 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.281389 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.283956 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.283962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.283972 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.283993 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.285559 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.285960 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.285965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.285970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.286070 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.286346 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.286349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.286351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.286848 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.286984 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.287289 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.287598 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260925 00:55:51.288631 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.279159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39313 (local address 127.0.0.1:33716)
0925 00:55:51.279623 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.279668 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.279795 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.280083 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.280087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.280426 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.280562 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.280568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.281328 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.281331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.281964 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.281972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.282443 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.282447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.282538 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.282997 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.283016 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.284580 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.286139 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.286147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.286154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.286372 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.286620 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.286623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.286624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.286703 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.287067 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.287072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.287205 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.287373 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.287680 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":93,"thread_start_us":177,"threads_started":1}
I20260925 00:55:51.289872 30714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:51.297879 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.290350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:34139 (local address 127.0.0.1:39060)
0925 00:55:51.290644 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.290657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.290735 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.290944 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.290946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.291094 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.291168 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.291173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.291894 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.291898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.292417 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.292424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.292923 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.292926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.293004 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.293426 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.293440 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.294784 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.296418 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.296422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.296424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.296644 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.296934 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.296938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.296940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.297039 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.297400 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.297403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.297506 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.297598 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.297708 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260925 00:55:51.297919 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.290482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.16.202.61:34139)
0925 00:55:51.290590 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.290595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.290710 (+ 115us) server_negotiation.cc:407] Connection header received
0925 00:55:51.290810 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.290814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.290861 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.290923 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.291273 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.291797 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.292521 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.292821 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.292914 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.294944 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.294948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.294950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.294976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.296311 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.296756 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.296760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.296762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.296848 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.297133 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.297136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.297137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.297316 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.297388 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.297677 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.297802 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 00:55:51.298796 30714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:55:51.298938 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.290981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.16.202.62:39313)
0925 00:55:51.291092 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.291096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.291146 (+ 50us) server_negotiation.cc:407] Connection header received
0925 00:55:51.291267 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.291270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.291307 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.291366 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.291976 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.292451 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.293252 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.293461 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.293544 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.295887 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.295890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.295893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.295915 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.297352 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.297777 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.297781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.297783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.297862 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.298130 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.298134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.298136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.298360 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.298588 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.298690 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.298805 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:55:51.301632 30563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } attempt: 3
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:55:51.305872 30638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } attempt: 3
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
I20260925 00:55:51.307665 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.299508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.16.202.60:37947)
0925 00:55:51.299626 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.299630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.299642 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:55:51.299673 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.299676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.299713 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.299773 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.300154 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.300691 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.301375 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.301605 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.301691 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.303759 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.303763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.303765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.303786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.305897 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.306547 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.306550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.306552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.306629 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.306901 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.306904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.306906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.307095 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.307176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.307407 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.307531 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":44}
I20260925 00:55:51.307665 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.299294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42768)
0925 00:55:51.299392 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.299406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.299476 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.299809 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.299812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.299960 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.300054 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.300060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.300772 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.300775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.301281 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.301288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.301790 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.301793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.301866 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.302279 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.302292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.303655 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.305989 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.305993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.305995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.306452 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.306727 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.306731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.306733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.306822 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.307205 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.307208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.307271 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.307348 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.307476 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
Sep 25 00:55:51 dist-test-slave-rpk1 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
I20260925 00:55:51.311226 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.300097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.16.202.61:34139)
0925 00:55:51.300199 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.300203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.300215 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:55:51.300326 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.300329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.300366 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.300428 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.301060 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.301529 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.302357 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.302609 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.302700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.305383 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.305386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.305389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.305415 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.309278 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.309835 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.309838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.309840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.309925 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.310209 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.310212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.310214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.310411 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.310611 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.310960 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.311076 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:55:51.312140 30714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: Bootstrap starting.
I20260925 00:55:51.315032 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:55:51.315068 30714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:51.315156 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.306247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42778)
0925 00:55:51.306344 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.306358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.306425 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.306770 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.306773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.306942 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.307037 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.307054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.308239 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.308242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.309059 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.309069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.309611 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.309615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.309697 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.310081 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.310094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.311391 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.313245 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.313248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.313250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.313480 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.314194 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.314197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.314199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.314286 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.314708 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.314711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.314802 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.314878 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.315003 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 00:55:51.315321 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.302563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:37947 (local address 127.0.0.1:42772)
0925 00:55:51.302742 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:51.302756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:51.302836 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:51.303276 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:51.303279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:51.303419 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:51.303517 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.303524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.304511 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.304515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.305271 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:51.305281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.305827 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.305831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:51.305910 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.306427 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:51.306440 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:51.307730 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:51.313496 (+ 5766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.313500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.313502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.313759 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.314263 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:51.314267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:51.314268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:51.314364 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:51.314855 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:51.314858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:51.314964 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 00:55:51.315046 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 00:55:51.315179 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 00:55:51.315680 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.306281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.16.202.60:37947)
0925 00:55:51.306553 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.306562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.306573 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:55:51.306600 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.306603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.306638 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.306751 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.307378 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.308145 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.309181 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.309502 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.309590 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.311525 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.311529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.311532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.311555 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.313128 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.313612 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.313615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.313617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.314090 (+ 473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.314390 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.314393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.314395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.314602 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.315322 (+ 720us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.315398 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.315524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20260925 00:55:51.315932 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.302321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.16.202.60:37947)
0925 00:55:51.302692 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.302701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.302800 (+ 99us) server_negotiation.cc:407] Connection header received
0925 00:55:51.302913 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.302916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.302965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.303034 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.303652 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.304416 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.305367 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.305723 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.305811 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.308373 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.308376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.308378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.308398 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.313368 (+ 4970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.313856 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.313858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.313859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.314155 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.314532 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.314535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.314536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.314739 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.315594 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.315669 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.315778 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":126,"threads_started":1}
I20260925 00:55:51.318660 30638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: Bootstrap starting.
I20260925 00:55:51.318801 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7: Bootstrap starting.
I20260925 00:55:51.321380 30563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:51.321390 30638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:51.322278 30638 log.cc:824] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:51.322283 30563 log.cc:824] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:51.322793 30714 log.cc:824] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:51.324425 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7: No bootstrap required, opened a new log
I20260925 00:55:51.325341 30714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: No bootstrap required, opened a new log
I20260925 00:55:51.325410 30638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: No bootstrap required, opened a new log
I20260925 00:55:51.328472 30563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.328747 30563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:51.328799 30563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f064c5dd34a43a9b8e2fc6894d92ea7, State: Initialized, Role: FOLLOWER
I20260925 00:55:51.328914 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.318042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.16.202.61:34139)
0925 00:55:51.318705 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.318709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.319748 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 00:55:51.319911 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.319914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.319960 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.320006 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.320903 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.321617 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.322666 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.322949 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.323023 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.324859 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.324863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.324865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.324889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.326420 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.326938 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.326941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.326943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.327022 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.327417 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.327421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.327423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.327757 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.327906 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.328341 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.328756 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590}
I20260925 00:55:51.329399 30638 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.329665 30638 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:51.329753 30638 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68a1051028444768518b834338a6fdb, State: Initialized, Role: FOLLOWER
I20260925 00:55:51.329849 30563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.330443 30638 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.331296 30563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:51.331887 30638 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:51.331957 30714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.332230 30714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:51.332293 30714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768a4154e64940e9a528504da71e6933, State: Initialized, Role: FOLLOWER
I20260925 00:55:51.332774 30714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.334053 30714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:51.335521 30732 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.335759 30732 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:55:51.337869 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.317373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.16.202.62:39313)
0925 00:55:51.317479 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.317483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.318380 (+ 897us) server_negotiation.cc:407] Connection header received
0925 00:55:51.318567 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.318570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.318615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.318664 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.319586 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.320330 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.321454 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.321757 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.321833 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.323640 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.323643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.323646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.323670 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.329309 (+ 5639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.336396 (+ 7087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.336400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.336402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.336498 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.336832 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.336836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.336837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.337070 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.337165 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.337581 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.337714 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:55:51.339166 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:51.318312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.16.202.60:37947)
0925 00:55:51.318792 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:51.318795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:51.330634 (+ 11839us) server_negotiation.cc:407] Connection header received
0925 00:55:51.330681 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:51.330684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:51.330735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:51.330804 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:51.331528 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.332253 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.333352 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:51.333701 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:51.333798 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:51.335400 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:51.335404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:51.335407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:51.335435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:51.337062 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.337537 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.337541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.337543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.337631 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:51.337909 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:51.337912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:51.337914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:51.338121 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:51.338834 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:51.338909 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:55:51.339024 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
W20260925 00:55:51.340399 30755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:55:51.340476 30755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:55:51.340806 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.341030 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:55:51.342839 30495 master_runner.cc:263] No leader master found from master 7f064c5dd34a43a9b8e2fc6894d92ea7
I20260925 00:55:51.342911 30495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:55:51.345391 30645 master_runner.cc:263] No leader master found from master 768a4154e64940e9a528504da71e6933
I20260925 00:55:51.345472 30645 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:55:51.345785 30570 master_runner.cc:263] No leader master found from master e68a1051028444768518b834338a6fdb
I20260925 00:55:51.345845 30570 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 00:55:51.348598 30748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:55:51.348852 30748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:55:51.355278 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.355453 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 00:55:51.355963 30766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:55:51.356125 30766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:55:51.371300 30731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:55:51.371527 30731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.372803 30731 leader_election.cc:288] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e68a1051028444768518b834338a6fdb (127.16.202.61:34139), 768a4154e64940e9a528504da71e6933 (127.16.202.60:37947)
I20260925 00:55:51.373639 30688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768a4154e64940e9a528504da71e6933" is_pre_election: true
I20260925 00:55:51.374094 30688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f064c5dd34a43a9b8e2fc6894d92ea7 in term 0.
I20260925 00:55:51.374925 30511 leader_election.cc:302] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768a4154e64940e9a528504da71e6933, 7f064c5dd34a43a9b8e2fc6894d92ea7; no voters:
I20260925 00:55:51.375902 30612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68a1051028444768518b834338a6fdb" is_pre_election: true
I20260925 00:55:51.376259 30612 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f064c5dd34a43a9b8e2fc6894d92ea7 in term 0.
I20260925 00:55:51.379631 30731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:55:51.379756 30731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:55:51.379801 30731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:51.381624 30731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.382592 30688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768a4154e64940e9a528504da71e6933"
I20260925 00:55:51.382737 30688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:51.384076 30688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f064c5dd34a43a9b8e2fc6894d92ea7 in term 1.
I20260925 00:55:51.384524 30511 leader_election.cc:302] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768a4154e64940e9a528504da71e6933, 7f064c5dd34a43a9b8e2fc6894d92ea7; no voters:
I20260925 00:55:51.385293 30612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68a1051028444768518b834338a6fdb"
I20260925 00:55:51.385435 30612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:51.386682 30612 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f064c5dd34a43a9b8e2fc6894d92ea7 in term 1.
I20260925 00:55:51.387099 30731 leader_election.cc:288] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [CANDIDATE]: Term 1 election: Requested vote from peers e68a1051028444768518b834338a6fdb (127.16.202.61:34139), 768a4154e64940e9a528504da71e6933 (127.16.202.60:37947)
I20260925 00:55:51.387315 30731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:51.387449 30731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [term 1 LEADER]: Becoming Leader. State: Replica: 7f064c5dd34a43a9b8e2fc6894d92ea7, State: Running, Role: LEADER
I20260925 00:55:51.387840 30731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } }
I20260925 00:55:51.389954 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f064c5dd34a43a9b8e2fc6894d92ea7. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.390127 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:51.393154 30772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:51.394169 30772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:51.405897 30688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [term 1 FOLLOWER]: Refusing update from remote peer 7f064c5dd34a43a9b8e2fc6894d92ea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:51.406859 30770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:51.412642 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f064c5dd34a43a9b8e2fc6894d92ea7. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.412832 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:55:51.414160 30612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [term 1 FOLLOWER]: Refusing update from remote peer 7f064c5dd34a43a9b8e2fc6894d92ea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:51.417508 30770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:51.422355 30732 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f064c5dd34a43a9b8e2fc6894d92ea7. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.422503 30732 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:55:51.429654 30732 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.429791 30732 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:55:51.432919 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.433071 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:51.433329 30773 mvcc.cc:205] Tried to move back new op lower bound from 7333059589736304640 to 7333059589688672256. Current Snapshot: MvccSnapshot[applied={T|T < 7333059589736304640}]
I20260925 00:55:51.435302 30776 mvcc.cc:205] Tried to move back new op lower bound from 7333059589736304640 to 7333059589688672256. Current Snapshot: MvccSnapshot[applied={T|T < 7333059589736304640}]
I20260925 00:55:51.436236 30770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.436359 30770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:51.437549 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f064c5dd34a43a9b8e2fc6894d92ea7" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39313 } } peers { permanent_uuid: "e68a1051028444768518b834338a6fdb" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 34139 } } peers { permanent_uuid: "768a4154e64940e9a528504da71e6933" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 37947 } } }
I20260925 00:55:51.437668 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:55:51.440415 30774 mvcc.cc:205] Tried to move back new op lower bound from 7333059589736304640 to 7333059589688672256. Current Snapshot: MvccSnapshot[applied={T|T < 7333059589736304640}]
I20260925 00:55:51.444602 30772 catalog_manager.cc:1347] Generated new cluster ID: 9ef7c49399d94001a5730d3f265f81b7
I20260925 00:55:51.444705 30772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:55:51.479423 30772 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:55:51.481036 30772 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:55:51.517091 30772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f064c5dd34a43a9b8e2fc6894d92ea7: Generated new TSK 0
I20260925 00:55:51.518217 30772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:55:52.341830 30755 catalog_manager.cc:1259] Loaded cluster ID: 9ef7c49399d94001a5730d3f265f81b7
I20260925 00:55:52.341914 30755 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: loading cluster ID for follower catalog manager: success
I20260925 00:55:52.343683 30755 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: acquiring CA information for follower catalog manager: success
I20260925 00:55:52.344997 30755 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e68a1051028444768518b834338a6fdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:55:52.357583 30766 catalog_manager.cc:1259] Loaded cluster ID: 9ef7c49399d94001a5730d3f265f81b7
I20260925 00:55:52.357673 30766 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: loading cluster ID for follower catalog manager: success
I20260925 00:55:52.359354 30766 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: acquiring CA information for follower catalog manager: success
I20260925 00:55:52.360553 30766 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 768a4154e64940e9a528504da71e6933: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:55:52.592417 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30495
I20260925 00:55:52.598134 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30570
I20260925 00:55:52.603224 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30645
2026-09-25T00:55:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3035 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:55:52 dist-test-slave-rpk1 krb5kdc[30786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:55:52 dist-test-slave-rpk1 krb5kdc[30786](info): set up 2 sockets
Sep 25 00:55:52 dist-test-slave-rpk1 krb5kdc[30786](info): commencing operation
krb5kdc: starting...
W20260925 00:55:53.560942 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:55:53 dist-test-slave-rpk1 krb5kdc[30786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:55:53.688386 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:44261
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:44261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:53.797165 30799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:53.797446 30799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:53.797487 30799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:53.801519 30799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:53.801594 30799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:53.801613 30799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:53.801628 30799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:53.801643 30799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:53.807070 30799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:44261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:53.808162 30799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:53.809274 30799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:53.817970 30807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:53.818176 30799 server_base.cc:1034] running on GCE node
W20260925 00:55:53.817979 30805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:53.818099 30804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:53.819025 30799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:53.820263 30799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:53.821517 30799 hybrid_clock.cc:648] HybridClock initialized: now 1790297753821493 us; error 42 us; skew 500 ppm
Sep 25 00:55:53 dist-test-slave-rpk1 krb5kdc[30786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:53.824616 30799 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:53.825613 30799 webserver.cc:460] Webserver started at http://127.16.202.62:33135/ using document root <none> and password file <none>
I20260925 00:55:53.826174 30799 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:53.826241 30799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:53.826435 30799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:53.828548 30799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "3d19ef0e665845bfa1c52712a7085354"
format_stamp: "Formatted at 2026-09-25 00:55:53 on dist-test-slave-rpk1"
server_key: "ece92e30b29bb2270610daa4bc9ca3fa"
server_key_iv: "280ef6f37004e50a3ec237655d7c3774"
server_key_version: "encryptionkey@0"
I20260925 00:55:53.829140 30799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "3d19ef0e665845bfa1c52712a7085354"
format_stamp: "Formatted at 2026-09-25 00:55:53 on dist-test-slave-rpk1"
server_key: "ece92e30b29bb2270610daa4bc9ca3fa"
server_key_iv: "280ef6f37004e50a3ec237655d7c3774"
server_key_version: "encryptionkey@0"
I20260925 00:55:53.833081 30799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:55:53.835781 30814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:53.837288 30799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:53.837428 30799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "3d19ef0e665845bfa1c52712a7085354"
format_stamp: "Formatted at 2026-09-25 00:55:53 on dist-test-slave-rpk1"
server_key: "ece92e30b29bb2270610daa4bc9ca3fa"
server_key_iv: "280ef6f37004e50a3ec237655d7c3774"
server_key_version: "encryptionkey@0"
I20260925 00:55:53.837556 30799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:53.854575 30799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:53.857875 30799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:53.858131 30799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:53.865077 30799 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:44261
I20260925 00:55:53.865114 30866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:44261 every 8 connection(s)
I20260925 00:55:53.866080 30799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:53.868981 30867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:53.874755 30867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354: Bootstrap starting.
I20260925 00:55:53.874954 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30799
I20260925 00:55:53.875049 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:55:53.875289 17192 external_mini_cluster.cc:1428] Setting key c6c3041a98b1980d2c3af08e96b689d0
I20260925 00:55:53.877401 30867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:53.878268 30867 log.cc:824] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:53.880179 30867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354: No bootstrap required, opened a new log
Sep 25 00:55:53 dist-test-slave-rpk1 krb5kdc[30786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:53.884086 30867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d19ef0e665845bfa1c52712a7085354" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44261 } }
I20260925 00:55:53.884380 30867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:53.884451 30867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d19ef0e665845bfa1c52712a7085354, State: Initialized, Role: FOLLOWER
I20260925 00:55:53.884970 30867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d19ef0e665845bfa1c52712a7085354" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44261 } }
I20260925 00:55:53.885205 30867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:55:53.885285 30867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:55:53.885392 30867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:53.886359 30867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d19ef0e665845bfa1c52712a7085354" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44261 } }
I20260925 00:55:53.886736 30867 leader_election.cc:302] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d19ef0e665845bfa1c52712a7085354; no voters:
I20260925 00:55:53.887121 30867 leader_election.cc:288] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:55:53.887221 30872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:53.887465 30872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [term 1 LEADER]: Becoming Leader. State: Replica: 3d19ef0e665845bfa1c52712a7085354, State: Running, Role: LEADER
I20260925 00:55:53.887828 30872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d19ef0e665845bfa1c52712a7085354" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44261 } }
I20260925 00:55:53.888589 30867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:53.889703 30873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d19ef0e665845bfa1c52712a7085354. Latest consensus state: current_term: 1 leader_uuid: "3d19ef0e665845bfa1c52712a7085354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d19ef0e665845bfa1c52712a7085354" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44261 } } }
I20260925 00:55:53.889756 30874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d19ef0e665845bfa1c52712a7085354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d19ef0e665845bfa1c52712a7085354" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44261 } } }
I20260925 00:55:53.889876 30874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:53.889987 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:53.876562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.16.202.62:44261)
0925 00:55:53.877005 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:53.877021 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:53.877057 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:55:53.877758 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:53.877784 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:53.878162 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:53.878474 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:53.879246 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:53.880206 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:53.881098 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:53.881563 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:53.881707 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:53.884497 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:53.884504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:53.884522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:53.884552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:53.886398 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:53.886952 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:53.886958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:53.886965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:53.887115 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:53.887405 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:53.887408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:53.887410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:53.887968 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:53.888181 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:53.888431 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 00:55:53.888924 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260925 00:55:53.889825 30873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354 [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:53.893560 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 00:55:53.894485 30887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:55:53.894567 30887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:55:53.894622 30881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:53.895476 30881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:53.900377 30881 catalog_manager.cc:1347] Generated new cluster ID: e02b61460bee4c9a8a6f35c3ea0c0faa
I20260925 00:55:53.900444 30881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:55:53.908932 30881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:55:53.909885 30881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:55:53.915246 30881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d19ef0e665845bfa1c52712a7085354: Generated new TSK 0
I20260925 00:55:53.915959 30881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:55:53.949950 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30799
2026-09-25T00:55:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1345 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:55:54 dist-test-slave-rpk1 krb5kdc[30894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:55:54 dist-test-slave-rpk1 krb5kdc[30894](info): set up 2 sockets
Sep 25 00:55:54 dist-test-slave-rpk1 krb5kdc[30894](info): commencing operation
krb5kdc: starting...
W20260925 00:55:54.908843 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:55:54 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:55:55.032302 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39987
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:55.143484 30907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:55.143797 30907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:55.143839 30907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:55.148277 30907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:55.148357 30907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:55.148376 30907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:55.148392 30907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:55.148411 30907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:55.153971 30907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:55.155055 30907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:55.156059 30907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:55.164582 30915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.164599 30912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.164605 30913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:55.164732 30907 server_base.cc:1034] running on GCE node
I20260925 00:55:55.165580 30907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:55.166647 30907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:55.167812 30907 hybrid_clock.cc:648] HybridClock initialized: now 1790297755167785 us; error 41 us; skew 500 ppm
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:55.175334 30907 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:55.176343 30907 webserver.cc:460] Webserver started at http://127.16.202.62:44065/ using document root <none> and password file <none>
I20260925 00:55:55.176965 30907 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:55.177033 30907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:55.177232 30907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:55.179304 30907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "87b204759c024529966622aae907aabc"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "843c0f96bcb6c98d812d09a860eff6fa"
server_key_iv: "ee9f91e7eccf0f795bb5af76a5b0fa70"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.179842 30907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "87b204759c024529966622aae907aabc"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "843c0f96bcb6c98d812d09a860eff6fa"
server_key_iv: "ee9f91e7eccf0f795bb5af76a5b0fa70"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.183748 30907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:55:55.186515 30922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.187870 30907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:55:55.187999 30907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "87b204759c024529966622aae907aabc"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "843c0f96bcb6c98d812d09a860eff6fa"
server_key_iv: "ee9f91e7eccf0f795bb5af76a5b0fa70"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.188115 30907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:55.210047 30907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:55.213245 30907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:55.213430 30907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:55.220151 30907 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39987
I20260925 00:55:55.220413 30974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39987 every 8 connection(s)
I20260925 00:55:55.221170 30907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:55.224081 30975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:55.228853 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 30907
I20260925 00:55:55.228953 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:55:55.229189 17192 external_mini_cluster.cc:1428] Setting key ae1625bc969ce3a7ab0723824ac5dcd0
I20260925 00:55:55.229969 30975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap starting.
I20260925 00:55:55.232271 30975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:55.233224 30975 log.cc:824] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:55.235334 30975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: No bootstrap required, opened a new log
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:55.239559 30975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:55.239863 30975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:55.239936 30975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b204759c024529966622aae907aabc, State: Initialized, Role: FOLLOWER
I20260925 00:55:55.240414 30975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:55.240549 30975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:55:55.240617 30975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:55:55.240742 30975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:55.241816 30975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:55.242314 30975 leader_election.cc:302] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b204759c024529966622aae907aabc; no voters:
I20260925 00:55:55.242703 30975 leader_election.cc:288] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:55:55.242784 30980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:55.242986 30980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 LEADER]: Becoming Leader. State: Replica: 87b204759c024529966622aae907aabc, State: Running, Role: LEADER
I20260925 00:55:55.243327 30980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:55.243695 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.230663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.16.202.62:39987)
0925 00:55:55.231083 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:55.231100 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:55.231135 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:55:55.231776 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:55.231793 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:55.232144 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:55.232464 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:55.233351 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.234349 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.235460 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.235958 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.236120 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.238781 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:55.238786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:55.238800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:55.238827 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:55.240505 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.241050 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.241055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.241060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.241175 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.241440 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.241441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.241443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.241876 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:55.242055 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:55.242375 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 00:55:55.242691 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260925 00:55:55.244096 30975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:55.245159 30982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b204759c024529966622aae907aabc. Latest consensus state: current_term: 1 leader_uuid: "87b204759c024529966622aae907aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } } }
I20260925 00:55:55.245309 30982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:55.245663 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b204759c024529966622aae907aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } } }
I20260925 00:55:55.245772 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: This master's current role is: LEADER
W20260925 00:55:55.249212 30995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:55:55.249281 30995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:55:55.249356 30989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:55.250046 30989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:55.255591 30989 catalog_manager.cc:1347] Generated new cluster ID: 5a1813a5dc184d1396efe8f7b58e92ac
I20260925 00:55:55.255678 30989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:55:55.273741 30989 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:55:55.274680 30989 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:55:55.281204 30989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Generated new TSK 0
I20260925 00:55:55.282068 30989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:55:55.316073 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39987
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:55.431133 31001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:55.431492 31001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:55.431548 31001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:55.435751 31001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:55.435833 31001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:55.435894 31001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:55:55.441520 31001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:55.442631 31001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:55.443643 31001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:55.453699 31006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.453699 31009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:55.453758 31001 server_base.cc:1034] running on GCE node
W20260925 00:55:55.453707 31007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:55.454293 31001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:55.454916 31001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:55.456099 31001 hybrid_clock.cc:648] HybridClock initialized: now 1790297755456067 us; error 42 us; skew 500 ppm
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:55.458907 31001 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:55:55.459949 31001 webserver.cc:460] Webserver started at http://127.16.202.1:39671/ using document root <none> and password file <none>
I20260925 00:55:55.460469 31001 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:55.460567 31001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:55.460798 31001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:55.462767 31001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "35258f82d3a140e59918df23289f3bd6"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "2d53d5d3640dc1c587fdaba954609e0f"
server_key_iv: "35b278b9fca6ab321cb6eba6fc33ca77"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.463271 31001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "35258f82d3a140e59918df23289f3bd6"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "2d53d5d3640dc1c587fdaba954609e0f"
server_key_iv: "35b278b9fca6ab321cb6eba6fc33ca77"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.466939 31001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:55:55.469491 31016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.470858 31001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:55.470981 31001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "35258f82d3a140e59918df23289f3bd6"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "2d53d5d3640dc1c587fdaba954609e0f"
server_key_iv: "35b278b9fca6ab321cb6eba6fc33ca77"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.471104 31001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:55.495697 31001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:55.498981 31001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:55.499163 31001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:55.499768 31001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:55.500636 31001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:55.500772 31001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.500865 31001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:55.500898 31001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.509500 31001 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:46485
I20260925 00:55:55.509553 31129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:46485 every 8 connection(s)
I20260925 00:55:55.510416 31001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:55:55.512786 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31001
I20260925 00:55:55.512917 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:55:55.513242 17192 external_mini_cluster.cc:1428] Setting key 0779fff94e27ebefadd781837e4ab425
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:55:55.524652 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.512195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:43437 (local address 127.16.202.62:39987)
0925 00:55:55.512356 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:55.512361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:55.513394 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 00:55:55.514867 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:55.514870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:55.514923 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:55.514995 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:55.516215 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.516812 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.518297 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.518534 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.518622 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.521193 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:55.521198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:55.521201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:55.521226 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:55.522540 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.523051 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.523053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.523055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.523123 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.523547 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.523560 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.523561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.523760 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:55.523845 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:55.524360 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 00:55:55.524488 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:55:55.525549 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.512701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:43437)
0925 00:55:55.513240 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:55.513287 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:55.514650 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:55.515174 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:55.515181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:55.515541 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:55.516064 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:55.516079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.516933 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.516936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:55.518170 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:55.518181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.518695 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.518699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:55.518927 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.519583 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:55.519599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:55.521026 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:55.522652 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:55.522658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:55.522677 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:55.522962 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:55.523201 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:55.523204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:55.523205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:55.523343 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:55.523877 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:55.523883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:55.524137 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 00:55:55.524296 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 00:55:55.524596 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260925 00:55:55.526851 31130 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39987
I20260925 00:55:55.527100 31130 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:55.527666 31130 heartbeater.cc:511] Master 127.16.202.62:39987 requested a full tablet report, sending...
I20260925 00:55:55.529335 30939 ts_manager.cc:194] Registered new tserver with Master: 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485)
I20260925 00:55:55.530910 30939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:43437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:55:55.541900 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39987
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:55.655380 31135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:55.655682 31135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:55.655756 31135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:55.659969 31135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:55.660049 31135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:55.660130 31135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:55:55.665804 31135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:55.666981 31135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:55.668026 31135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:55:55.678118 31135 server_base.cc:1034] running on GCE node
W20260925 00:55:55.678077 31143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.678077 31140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.678074 31141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:55.678872 31135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:55.679777 31135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:55.680966 31135 hybrid_clock.cc:648] HybridClock initialized: now 1790297755680936 us; error 51 us; skew 500 ppm
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:55.686934 31135 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:55:55.688127 31135 webserver.cc:460] Webserver started at http://127.16.202.2:34087/ using document root <none> and password file <none>
I20260925 00:55:55.688829 31135 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:55.688911 31135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:55.689121 31135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:55.691214 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3c464da8d5604da55a4100016b90cb3f"
server_key_iv: "3679d40e3cc61ccf8dd09f4461750c0f"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.691782 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3c464da8d5604da55a4100016b90cb3f"
server_key_iv: "3679d40e3cc61ccf8dd09f4461750c0f"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.695919 31135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:55:55.698742 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.700114 31135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:55:55.700229 31135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3c464da8d5604da55a4100016b90cb3f"
server_key_iv: "3679d40e3cc61ccf8dd09f4461750c0f"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.700348 31135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:55.715845 31135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:55.719273 31135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:55.719435 31135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:55.720065 31135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:55.720947 31135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:55.720999 31135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.721037 31135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:55.721055 31135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.730105 31135 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:37639
I20260925 00:55:55.730124 31263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:37639 every 8 connection(s)
I20260925 00:55:55.731022 31135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:55:55.738842 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31135
I20260925 00:55:55.738945 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:55:55.739192 17192 external_mini_cluster.cc:1428] Setting key 166c6782ff4a678f706b2a2b41bae115
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:55.745077 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.732921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:32903 (local address 127.16.202.62:39987)
0925 00:55:55.733059 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:55.733063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:55.734003 (+ 940us) server_negotiation.cc:407] Connection header received
0925 00:55:55.735357 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:55.735360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:55.735419 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:55.735500 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:55.736746 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.737283 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.738397 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.738643 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.738732 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.741289 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:55.741292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:55.741294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:55.741318 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:55.742779 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.743407 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.743410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.743411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.743495 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.743828 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.743829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.743831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.744052 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:55.744128 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:55.744752 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 00:55:55.744872 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:55:55.746328 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.733344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:32903)
0925 00:55:55.733828 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:55.733874 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:55.735140 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:55.735696 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:55.735704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:55.736083 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:55.736581 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:55.736595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.737405 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.737408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:55.738278 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:55.738287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.738787 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.738791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:55.738977 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.739566 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:55.739581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:55.741128 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:55.742906 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:55.742913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:55.742954 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:55.743310 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:55.743573 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:55.743576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:55.743577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:55.743749 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:55.744144 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:55.744151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:55.744468 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 00:55:55.744708 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 00:55:55.745160 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260925 00:55:55.747639 31264 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39987
I20260925 00:55:55.747920 31264 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:55.748554 31264 heartbeater.cc:511] Master 127.16.202.62:39987 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:55:55.749795 30939 ts_manager.cc:194] Registered new tserver with Master: f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639)
I20260925 00:55:55.750388 30939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:32903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:55:55.769393 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39987
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:55.896082 31269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:55.896377 31269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:55.896422 31269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:55.900620 31269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:55.900722 31269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:55.900784 31269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:55:55.906474 31269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:55.907598 31269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:55.908740 31269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:55.918814 31274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.918820 31275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:55.918820 31277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:55.918959 31269 server_base.cc:1034] running on GCE node
I20260925 00:55:55.919538 31269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:55.920200 31269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:55.921376 31269 hybrid_clock.cc:648] HybridClock initialized: now 1790297755921356 us; error 34 us; skew 500 ppm
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:55.924171 31269 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:55:55.925146 31269 webserver.cc:460] Webserver started at http://127.16.202.3:44647/ using document root <none> and password file <none>
I20260925 00:55:55.925697 31269 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:55.925765 31269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:55.926008 31269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:55:55.928148 31269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "cc74147cae58406a9be23feace042daa"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3687cde9c2b6c224740ea7a5af252542"
server_key_iv: "7b31e812ca5018e224e465f9af83a7aa"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.928741 31269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "cc74147cae58406a9be23feace042daa"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3687cde9c2b6c224740ea7a5af252542"
server_key_iv: "7b31e812ca5018e224e465f9af83a7aa"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.932564 31269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:55:55.935177 31284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.936596 31269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:55:55.936735 31269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "cc74147cae58406a9be23feace042daa"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3687cde9c2b6c224740ea7a5af252542"
server_key_iv: "7b31e812ca5018e224e465f9af83a7aa"
server_key_version: "encryptionkey@0"
I20260925 00:55:55.936851 31269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:55.960294 31269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:55.964174 31269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:55.964382 31269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:55.965076 31269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:55.966064 31269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:55.966115 31269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.966169 31269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:55.966197 31269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:55.975238 31269 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:40813
I20260925 00:55:55.975288 31397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:40813 every 8 connection(s)
I20260925 00:55:55.976167 31269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:55:55.977305 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31269
I20260925 00:55:55.977399 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:55:55.977653 17192 external_mini_cluster.cc:1428] Setting key 1cade7c3e89ce80e5e248d8f850f0f68
Sep 25 00:55:55 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:55.990648 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.978053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:47265 (local address 127.16.202.62:39987)
0925 00:55:55.978176 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:55.978179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:55.979254 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 00:55:55.980646 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:55.980648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:55.980735 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:55.980814 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:55.982106 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.982680 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.983973 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.984202 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.984298 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.987012 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:55.987015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:55.987017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:55.987045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:55.988375 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.989066 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.989069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.989071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.989151 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:55.989573 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:55.989576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:55.989577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:55.989760 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:55.989834 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:55.990387 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 00:55:55.990513 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:55:55.991719 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:55.978595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:47265)
0925 00:55:55.979089 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:55.979161 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:55.980433 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:55.980990 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:55.980997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:55.981423 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:55.981940 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:55.981954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.982803 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.982806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:55.983847 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:55.983858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:55.984417 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:55.984421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:55.984591 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:55.985220 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:55.985238 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:55.986812 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:55.988528 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:55.988534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:55.988549 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:55.988966 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:55.989254 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:55.989257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:55.989258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:55.989461 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:55.989875 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:55.989881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:55.990156 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 00:55:55.990329 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 00:55:55.990732 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260925 00:55:55.993047 31398 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39987
I20260925 00:55:55.993273 31398 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:55.993768 31398 heartbeater.cc:511] Master 127.16.202.62:39987 requested a full tablet report, sending...
I20260925 00:55:55.994838 30939 ts_manager.cc:194] Registered new tserver with Master: cc74147cae58406a9be23feace042daa (127.16.202.3:40813)
I20260925 00:55:55.995416 30939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:47265
I20260925 00:55:56.003890 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:55:56.013577 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.006026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.16.202.62:39987)
0925 00:55:56.006141 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.006145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.006263 (+ 118us) server_negotiation.cc:407] Connection header received
0925 00:55:56.006356 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.006358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.006403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.006469 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:56.007169 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.007741 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.008666 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.008937 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.009030 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.010400 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:56.010403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:56.010405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:56.010433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:56.011849 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.012229 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.012232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.012233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.012312 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.012575 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.012577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.012578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.012857 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:56.012969 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.013298 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.013406 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:55:56.016894 30939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:55:56.020016 30939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:55:56.034571 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.028934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.16.202.3:40813)
0925 00:55:56.029385 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.029396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.029420 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:55:56.029499 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.029505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.029726 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.029889 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:56.030406 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.031568 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.032939 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.033834 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.033944 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.034107 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.034301 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.034431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":140,"threads_started":1}
I20260925 00:55:56.034564 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.029033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46485 (local address 127.0.0.1:45572)
0925 00:55:56.029642 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.029654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.029738 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.030420 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.030423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.030441 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:56.030533 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.030539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.031803 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.031807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.033046 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.033057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.034055 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.034059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.034151 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.034177 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.034235 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.034388 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":43,"thread_start_us":101,"threads_started":1}
I20260925 00:55:56.034564 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.028766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:40813 (local address 127.0.0.1:36692)
0925 00:55:56.029129 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.029184 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.029302 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.030103 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.030110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.030135 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:56.030275 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.030283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.031698 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.031701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.032787 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.032796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.033990 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.033993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.034108 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.034151 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.034255 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.034403 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":116,"thread_start_us":109,"threads_started":1}
I20260925 00:55:56.035274 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.029591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37639 (local address 127.0.0.1:51212)
0925 00:55:56.030605 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.030617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.030702 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.031200 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.031204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.031222 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:56.031301 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.031307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.032658 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.032662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.033820 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.033831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.034893 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.034896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.034983 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.035006 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.035069 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.035144 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":6,"thread_start_us":84,"threads_started":1}
I20260925 00:55:56.035332 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.029987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.16.202.2:37639)
0925 00:55:56.030377 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.030388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.030791 (+ 403us) server_negotiation.cc:407] Connection header received
0925 00:55:56.030868 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.030874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.031057 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.031199 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:56.031415 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.032507 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.033955 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.034771 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.034882 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.035038 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.035101 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.035197 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":105,"threads_started":1}
I20260925 00:55:56.035770 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.029596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.16.202.1:46485)
0925 00:55:56.029968 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.029975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.029995 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:55:56.030069 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.030075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.030272 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.030681 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:56.030822 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.031653 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.033177 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.033907 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.034984 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.035290 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.035571 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.035661 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":95,"threads_started":1}
I20260925 00:55:56.038265 31198 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbb3104e9eb403984013b18c69384db (DEFAULT_TABLE table=test-table [id=314724573d324a8b88885a3b4d79d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:56.038429 31064 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbb3104e9eb403984013b18c69384db (DEFAULT_TABLE table=test-table [id=314724573d324a8b88885a3b4d79d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:56.038301 31332 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbb3104e9eb403984013b18c69384db (DEFAULT_TABLE table=test-table [id=314724573d324a8b88885a3b4d79d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:56.039273 31198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbb3104e9eb403984013b18c69384db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:56.039337 31064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbb3104e9eb403984013b18c69384db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:56.039345 31332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbb3104e9eb403984013b18c69384db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:56.046088 31415 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Bootstrap starting.
I20260925 00:55:56.047351 31416 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Bootstrap starting.
I20260925 00:55:56.048838 31415 tablet_bootstrap.cc:654] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:56.049069 31417 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Bootstrap starting.
I20260925 00:55:56.049312 31416 tablet_bootstrap.cc:654] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:56.049724 31415 log.cc:824] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:56.050151 31416 log.cc:824] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:56.051738 31417 tablet_bootstrap.cc:654] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:56.051899 31415 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: No bootstrap required, opened a new log
I20260925 00:55:56.052047 31416 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: No bootstrap required, opened a new log
I20260925 00:55:56.052126 31415 ts_tablet_manager.cc:1397] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:55:56.052210 31416 ts_tablet_manager.cc:1397] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:55:56.052521 31417 log.cc:824] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:56.054203 31417 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: No bootstrap required, opened a new log
I20260925 00:55:56.054389 31417 ts_tablet_manager.cc:1397] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:55:56.056169 31415 raft_consensus.cc:348] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.056496 31415 raft_consensus.cc:374] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:56.056563 31415 raft_consensus.cc:729] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7984fab6cdc4dfcab6dad02799a1cda, State: Initialized, Role: FOLLOWER
I20260925 00:55:56.056594 31416 raft_consensus.cc:348] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.056857 31416 raft_consensus.cc:374] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:56.056915 31416 raft_consensus.cc:729] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35258f82d3a140e59918df23289f3bd6, State: Initialized, Role: FOLLOWER
I20260925 00:55:56.057085 31415 consensus_queue.cc:260] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.057296 31416 consensus_queue.cc:260] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.057832 31264 heartbeater.cc:503] Master 127.16.202.62:39987 was elected leader, sending a full tablet report...
I20260925 00:55:56.057753 31417 raft_consensus.cc:348] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.057958 31417 raft_consensus.cc:374] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:56.058012 31417 raft_consensus.cc:729] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc74147cae58406a9be23feace042daa, State: Initialized, Role: FOLLOWER
I20260925 00:55:56.058192 31416 ts_tablet_manager.cc:1428] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:55:56.058386 31417 consensus_queue.cc:260] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.058544 31130 heartbeater.cc:503] Master 127.16.202.62:39987 was elected leader, sending a full tablet report...
I20260925 00:55:56.059238 31398 heartbeater.cc:503] Master 127.16.202.62:39987 was elected leader, sending a full tablet report...
I20260925 00:55:56.060107 31415 ts_tablet_manager.cc:1428] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 00:55:56.062040 31417 ts_tablet_manager.cc:1428] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 00:55:56.133850 31422 raft_consensus.cc:482] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:55:56.134109 31422 raft_consensus.cc:504] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.135290 31422 leader_election.cc:288] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc74147cae58406a9be23feace042daa (127.16.202.3:40813), f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639)
I20260925 00:55:56.139516 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.135773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:33769 (local address 127.16.202.3:40813)
0925 00:55:56.135952 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.135957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.136127 (+ 170us) server_negotiation.cc:407] Connection header received
0925 00:55:56.136280 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.136284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.136335 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.136416 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:56.136763 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.137378 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.138535 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.138960 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.139070 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.139124 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.139314 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.139385 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 00:55:56.139536 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.136244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:39819 (local address 127.16.202.2:37639)
0925 00:55:56.136362 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.136366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.136528 (+ 162us) server_negotiation.cc:407] Connection header received
0925 00:55:56.136619 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.136622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.136708 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.136782 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:56.137010 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.137587 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.138574 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.138961 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.139067 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.139122 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.139336 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.139417 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:55:56.139566 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.135636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:40813 (local address 127.16.202.1:33769)
0925 00:55:56.136013 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.136026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.136169 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.136446 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.136449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.136470 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:56.136615 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.136622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.137489 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.137493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.138401 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.138411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.139073 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.139077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.139195 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.139221 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.139277 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.139348 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":4,"thread_start_us":144,"threads_started":1}
I20260925 00:55:56.139550 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.136103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37639 (local address 127.16.202.1:39819)
0925 00:55:56.136439 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.136452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.136542 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.136787 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.136789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.136805 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:56.136909 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.136914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.137696 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.137698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.138452 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.138461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.139078 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.139082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.139224 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.139241 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.139311 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.139381 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260925 00:55:56.140372 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb3104e9eb403984013b18c69384db" candidate_uuid: "35258f82d3a140e59918df23289f3bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc74147cae58406a9be23feace042daa" is_pre_election: true
I20260925 00:55:56.140460 31217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb3104e9eb403984013b18c69384db" candidate_uuid: "35258f82d3a140e59918df23289f3bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" is_pre_election: true
I20260925 00:55:56.140723 31352 raft_consensus.cc:2454] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35258f82d3a140e59918df23289f3bd6 in term 0.
I20260925 00:55:56.140769 31217 raft_consensus.cc:2454] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35258f82d3a140e59918df23289f3bd6 in term 0.
I20260925 00:55:56.141217 31020 leader_election.cc:302] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35258f82d3a140e59918df23289f3bd6, cc74147cae58406a9be23feace042daa; no voters:
I20260925 00:55:56.141499 31422 raft_consensus.cc:2781] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:55:56.141580 31422 raft_consensus.cc:482] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:55:56.141611 31422 raft_consensus.cc:3037] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:56.142678 31422 raft_consensus.cc:504] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.143023 31422 leader_election.cc:288] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 1 election: Requested vote from peers cc74147cae58406a9be23feace042daa (127.16.202.3:40813), f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639)
I20260925 00:55:56.143451 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb3104e9eb403984013b18c69384db" candidate_uuid: "35258f82d3a140e59918df23289f3bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc74147cae58406a9be23feace042daa"
I20260925 00:55:56.143607 31352 raft_consensus.cc:3037] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:56.143625 31217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb3104e9eb403984013b18c69384db" candidate_uuid: "35258f82d3a140e59918df23289f3bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
I20260925 00:55:56.143767 31217 raft_consensus.cc:3037] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:56.144827 31352 raft_consensus.cc:2454] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35258f82d3a140e59918df23289f3bd6 in term 1.
I20260925 00:55:56.144861 31217 raft_consensus.cc:2454] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35258f82d3a140e59918df23289f3bd6 in term 1.
I20260925 00:55:56.145355 31020 leader_election.cc:302] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35258f82d3a140e59918df23289f3bd6, cc74147cae58406a9be23feace042daa; no voters:
I20260925 00:55:56.145599 31422 raft_consensus.cc:2781] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:56.145849 31422 raft_consensus.cc:686] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 35258f82d3a140e59918df23289f3bd6, State: Running, Role: LEADER
I20260925 00:55:56.146250 31422 consensus_queue.cc:237] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.149657 30938 catalog_manager.cc:5495] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35258f82d3a140e59918df23289f3bd6 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "35258f82d3a140e59918df23289f3bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } health_report { overall_health: HEALTHY } } }
I20260925 00:55:56.163803 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.160366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.16.202.1:46485)
0925 00:55:56.160504 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.160509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.160524 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:55:56.160660 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.160663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.160742 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.160833 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:55:56.161072 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.161580 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.162408 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.162623 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.162708 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.163000 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:55:56.163428 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:55:56.163535 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.163602 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.163664 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:55:56.170018 31352 raft_consensus.cc:1261] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Refusing update from remote peer 35258f82d3a140e59918df23289f3bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:56.170018 31217 raft_consensus.cc:1261] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Refusing update from remote peer 35258f82d3a140e59918df23289f3bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:56.170795 31426 consensus_queue.cc:1035] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:56.170991 31422 consensus_queue.cc:1035] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:56.180018 31431 mvcc.cc:205] Tried to move back new op lower bound from 7333059609264422912 to 7333059609177931776. Current Snapshot: MvccSnapshot[applied={T|T < 7333059609264422912}]
I20260925 00:55:56.188234 30937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:56.188405 30937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:56.191066 30937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Sending DeleteTablet for 3 replicas of tablet 3fbb3104e9eb403984013b18c69384db
I20260925 00:55:56.192140 31332 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb3104e9eb403984013b18c69384db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:56 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36692
I20260925 00:55:56.192752 31198 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb3104e9eb403984013b18c69384db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:56 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:51212
I20260925 00:55:56.192739 31064 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb3104e9eb403984013b18c69384db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:56 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:45572
I20260925 00:55:56.193459 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31001
I20260925 00:55:56.194914 31440 tablet_replica.cc:330] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: stopping tablet replica
I20260925 00:55:56.195251 31439 tablet_replica.cc:330] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: stopping tablet replica
I20260925 00:55:56.195329 31440 raft_consensus.cc:2229] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:56.195570 31439 raft_consensus.cc:2229] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:56.195729 31440 raft_consensus.cc:2258] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:56.195884 31439 raft_consensus.cc:2258] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:56.197319 31440 ts_tablet_manager.cc:1905] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:55:56.197346 31439 ts_tablet_manager.cc:1905] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:55:56.199679 31439 ts_tablet_manager.cc:1918] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:56.199782 31439 log.cc:1196] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/3fbb3104e9eb403984013b18c69384db
I20260925 00:55:56.200008 31440 ts_tablet_manager.cc:1918] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:56.200073 31439 ts_tablet_manager.cc:1939] T 3fbb3104e9eb403984013b18c69384db P cc74147cae58406a9be23feace042daa: Deleting consensus metadata
I20260925 00:55:56.200119 31440 log.cc:1196] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/3fbb3104e9eb403984013b18c69384db
I20260925 00:55:56.200402 31440 ts_tablet_manager.cc:1939] T 3fbb3104e9eb403984013b18c69384db P f7984fab6cdc4dfcab6dad02799a1cda: Deleting consensus metadata
W20260925 00:55:56.201521 30924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:46485 (error 108)
I20260925 00:55:56.201576 30926 catalog_manager.cc:4841] TS f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639): tablet 3fbb3104e9eb403984013b18c69384db (table test-table [id=314724573d324a8b88885a3b4d79d0f1]) successfully deleted
I20260925 00:55:56.201782 30926 catalog_manager.cc:4841] TS cc74147cae58406a9be23feace042daa (127.16.202.3:40813): tablet 3fbb3104e9eb403984013b18c69384db (table test-table [id=314724573d324a8b88885a3b4d79d0f1]) successfully deleted
I20260925 00:55:56.201798 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31135
I20260925 00:55:56.202901 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.202599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46485 (local address 127.0.0.1:45602)
0925 00:55:56.202741 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.202817 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 00:55:56.203154 30924 catalog_manager.cc:4584] TS 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fbb3104e9eb403984013b18c69384db: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46485: connect: Connection refused (error 111)
I20260925 00:55:56.206959 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31269
I20260925 00:55:56.211519 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 30907
I20260925 00:55:56.217370 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39987
--webserver_interface=127.16.202.62
--webserver_port=44065
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:56.332309 31442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:56.332607 31442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:56.332648 31442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:56.336894 31442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:56.336975 31442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:56.336994 31442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:56.337010 31442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:56.337025 31442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:56.342600 31442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:56.343741 31442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:56.344832 31442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:56.358090 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.358089 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.358089 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:56.358376 31442 server_base.cc:1034] running on GCE node
I20260925 00:55:56.359061 31442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:56.360304 31442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:56.361509 31442 hybrid_clock.cc:648] HybridClock initialized: now 1790297756361471 us; error 62 us; skew 500 ppm
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:56.364289 31442 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:56.365392 31442 webserver.cc:460] Webserver started at http://127.16.202.62:44065/ using document root <none> and password file <none>
I20260925 00:55:56.365985 31442 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:56.366074 31442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:56.370260 31442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:55:56.372692 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.374143 31442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:56.374289 31442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "87b204759c024529966622aae907aabc"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "843c0f96bcb6c98d812d09a860eff6fa"
server_key_iv: "ee9f91e7eccf0f795bb5af76a5b0fa70"
server_key_version: "encryptionkey@0"
I20260925 00:55:56.374938 31442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:56.397110 31442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:56.400036 31442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:56.400238 31442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:56.407569 31442 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39987
I20260925 00:55:56.407893 31510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39987 every 8 connection(s)
I20260925 00:55:56.408746 31442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:56.413276 31511 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:55:56.414484 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31442
I20260925 00:55:56.414939 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:46485
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:39987
--webserver_port=39671
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:56.415287 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap starting.
I20260925 00:55:56.432590 31511 log.cc:824] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:56.441229 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:55:56.441612 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap complete.
I20260925 00:55:56.445633 31511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:56.446048 31511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b204759c024529966622aae907aabc, State: Initialized, Role: FOLLOWER
I20260925 00:55:56.446523 31511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:56.446640 31511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:55:56.446707 31511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:55:56.446801 31511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:55:56.448803 31511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:56.449154 31511 leader_election.cc:302] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b204759c024529966622aae907aabc; no voters:
I20260925 00:55:56.449461 31511 leader_election.cc:288] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:55:56.449568 31515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:55:56.449765 31515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 LEADER]: Becoming Leader. State: Replica: 87b204759c024529966622aae907aabc, State: Running, Role: LEADER
I20260925 00:55:56.450075 31515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:56.450790 31511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:56.451056 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b204759c024529966622aae907aabc. Latest consensus state: current_term: 2 leader_uuid: "87b204759c024529966622aae907aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } } }
I20260925 00:55:56.451166 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:56.451509 31519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:56.451643 31516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "87b204759c024529966622aae907aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } } }
I20260925 00:55:56.451882 31516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:56.456400 31519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:56.457132 31519 catalog_manager.cc:1259] Loaded cluster ID: 5a1813a5dc184d1396efe8f7b58e92ac
I20260925 00:55:56.457186 31519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:55:56.458389 31519 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 00:55:56.461177 31519 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 00:55:56.465655 31519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:55:56.466346 31519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Loaded TSK: 0
I20260925 00:55:56.467337 31519 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:55:56.537094 31513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:56.537379 31513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:56.537447 31513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:56.541625 31513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:56.541702 31513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:56.541782 31513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:55:56.547230 31513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:46485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=39671
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:56.548297 31513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:56.549321 31513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:56.559593 31538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.559692 31541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.559593 31539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:56.559998 31513 server_base.cc:1034] running on GCE node
I20260925 00:55:56.560501 31513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:56.561208 31513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:56.562393 31513 hybrid_clock.cc:648] HybridClock initialized: now 1790297756562367 us; error 55 us; skew 500 ppm
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:56.565325 31513 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:55:56.566524 31513 webserver.cc:460] Webserver started at http://127.16.202.1:39671/ using document root <none> and password file <none>
I20260925 00:55:56.567265 31513 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:56.567381 31513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:56.572921 31513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:55:56.575801 31548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.577526 31513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:56.577670 31513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "35258f82d3a140e59918df23289f3bd6"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "2d53d5d3640dc1c587fdaba954609e0f"
server_key_iv: "35b278b9fca6ab321cb6eba6fc33ca77"
server_key_version: "encryptionkey@0"
I20260925 00:55:56.578429 31513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:56.593453 31513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:56.597446 31513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:56.597657 31513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:56.598378 31513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:56.599787 31555 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:55:56.604436 31513 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:55:56.604542 31513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 00:55:56.604637 31513 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:55:56.607020 31513 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:55:56.607087 31513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 00:55:56.607427 31555 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Bootstrap starting.
I20260925 00:55:56.617159 31513 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:46485
I20260925 00:55:56.617242 31662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:46485 every 8 connection(s)
I20260925 00:55:56.618144 31513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:55:56.622074 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31513
I20260925 00:55:56.622571 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:37639
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:39987
--webserver_port=34087
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:56.627153 31555 log.cc:824] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:56.635730 31555 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:55:56.636135 31555 tablet_bootstrap.cc:492] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Bootstrap complete.
I20260925 00:55:56.637439 31555 ts_tablet_manager.cc:1397] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.015s
I20260925 00:55:56.637457 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.620101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:53241)
0925 00:55:56.620774 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.620830 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.621664 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.623250 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.623258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.623628 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:56.624182 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.624203 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.626276 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.626279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.627211 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.627223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.628190 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.628195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.628414 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.629238 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:56.629261 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:56.631206 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:56.633847 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:56.633854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:56.633870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:56.634248 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:56.634666 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:56.634669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:56.634672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:56.634855 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:56.635554 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:56.635562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:56.635866 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.636074 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.636422 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":147,"threads_started":1}
I20260925 00:55:56.637459 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.620188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:53241 (local address 127.16.202.62:39987)
0925 00:55:56.620621 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.620640 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.620953 (+ 313us) server_negotiation.cc:407] Connection header received
0925 00:55:56.622092 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.622103 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.622484 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.622826 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:56.624408 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.626087 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.627363 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.628030 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.628205 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.631464 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:56.631471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:56.631487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:56.631525 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:56.633696 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.634383 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.634390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.634395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.634560 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.634958 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.634961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.634963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.635442 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:56.635584 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.636298 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.636613 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260925 00:55:56.640165 31555 raft_consensus.cc:348] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.640640 31663 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39987
I20260925 00:55:56.640727 31555 raft_consensus.cc:729] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35258f82d3a140e59918df23289f3bd6, State: Initialized, Role: FOLLOWER
I20260925 00:55:56.640934 31663 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:56.641207 31555 consensus_queue.cc:260] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:56.641527 31663 heartbeater.cc:511] Master 127.16.202.62:39987 requested a full tablet report, sending...
I20260925 00:55:56.642181 31555 ts_tablet_manager.cc:1428] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260925 00:55:56.644340 31475 ts_manager.cc:194] Registered new tserver with Master: 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485)
I20260925 00:55:56.648109 31475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:53241
I20260925 00:55:56.650444 31663 heartbeater.cc:503] Master 127.16.202.62:39987 was elected leader, sending a full tablet report...
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:55:56.658286 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.647274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46485 (local address 127.0.0.1:45608)
0925 00:55:56.647698 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.647747 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.647882 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.648715 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.648721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.649115 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:56.649249 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.649259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.650560 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.650564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.651421 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.651431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.652487 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.652490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.652577 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.653160 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:56.653172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:56.654512 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:56.656526 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:56.656534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:56.656539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:56.656844 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:56.657162 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:56.657164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:56.657166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:56.657245 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:56.657750 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:56.657755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:56.657866 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.657976 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.658149 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260925 00:55:56.658776 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.647364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.16.202.1:46485)
0925 00:55:56.648095 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.648106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.648130 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:55:56.648317 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.648322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.648530 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.648667 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:56.649367 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.650430 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.651585 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.652365 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.652463 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.654652 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:56.654663 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:56.654677 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:56.654705 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:56.656409 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.656960 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.656967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.656968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.657069 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.657332 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.657334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.657336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.657634 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:56.657757 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.658087 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.658260 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":120,"threads_started":1}
I20260925 00:55:56.659754 31597 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb3104e9eb403984013b18c69384db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:56 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260925 00:55:56.660821 31672 tablet_replica.cc:330] stopping tablet replica
I20260925 00:55:56.661115 31672 raft_consensus.cc:2229] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:56.661353 31672 raft_consensus.cc:2258] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:56.662313 31672 ts_tablet_manager.cc:1905] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:55:56.666133 31672 ts_tablet_manager.cc:1918] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:56.666231 31672 log.cc:1196] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/3fbb3104e9eb403984013b18c69384db
I20260925 00:55:56.666451 31672 ts_tablet_manager.cc:1939] T 3fbb3104e9eb403984013b18c69384db P 35258f82d3a140e59918df23289f3bd6: Deleting consensus metadata
I20260925 00:55:56.668170 31460 catalog_manager.cc:4841] TS 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485): tablet 3fbb3104e9eb403984013b18c69384db (table test-table [id=314724573d324a8b88885a3b4d79d0f1]) successfully deleted
W20260925 00:55:56.743994 31667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:56.744277 31667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:56.744320 31667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:56.748483 31667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:56.748569 31667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:56.748629 31667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:55:56.754251 31667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:37639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=34087
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:56.755385 31667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:56.756359 31667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:56.766654 31678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.766530 31677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.766528 31680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:56.766889 31667 server_base.cc:1034] running on GCE node
I20260925 00:55:56.767354 31667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:56.768086 31667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:56.769268 31667 hybrid_clock.cc:648] HybridClock initialized: now 1790297756769236 us; error 48 us; skew 500 ppm
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:56.772158 31667 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:55:56.773149 31667 webserver.cc:460] Webserver started at http://127.16.202.2:34087/ using document root <none> and password file <none>
I20260925 00:55:56.773701 31667 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:56.773782 31667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:56.777838 31667 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:55:56.780130 31687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.781529 31667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:56.781677 31667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3c464da8d5604da55a4100016b90cb3f"
server_key_iv: "3679d40e3cc61ccf8dd09f4461750c0f"
server_key_version: "encryptionkey@0"
I20260925 00:55:56.782262 31667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:56.792729 31667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:56.795853 31667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:56.796046 31667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:56.796635 31667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:56.797519 31667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:56.797567 31667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.797621 31667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:56.797647 31667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.806701 31667 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:37639
I20260925 00:55:56.806733 31800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:37639 every 8 connection(s)
I20260925 00:55:56.807771 31667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:55:56.811087 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31667
I20260925 00:55:56.811519 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:40813
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:39987
--webserver_port=44647
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:56.822788 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.809840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:55407 (local address 127.16.202.62:39987)
0925 00:55:56.810009 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.810018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.810964 (+ 946us) server_negotiation.cc:407] Connection header received
0925 00:55:56.812748 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.812753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.812809 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.812902 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:56.814222 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.814982 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.816199 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.816564 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.816705 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.819157 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:56.819161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:56.819165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:56.819198 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:56.820798 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.821292 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.821296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.821298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.821391 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:56.821684 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:56.821688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:56.821689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:56.821938 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:56.822037 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:56.822481 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 00:55:56.822617 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 00:55:56.823627 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.810267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:55407)
0925 00:55:56.810772 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.810835 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.812433 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.813146 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.813155 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.813548 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:56.814063 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.814077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.815105 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.815107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.816072 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.816082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:56.816742 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:56.816745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:56.816897 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:56.817498 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:56.817515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:56.818967 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:56.820922 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:56.820927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:56.820941 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:56.821200 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:56.821470 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:56.821472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:56.821473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:56.821604 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:56.822022 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:56.822028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:56.822271 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 00:55:56.822426 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 00:55:56.822718 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260925 00:55:56.824734 31801 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39987
I20260925 00:55:56.824959 31801 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:56.825469 31801 heartbeater.cc:511] Master 127.16.202.62:39987 requested a full tablet report, sending...
I20260925 00:55:56.826507 31475 ts_manager.cc:194] Registered new tserver with Master: f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639)
I20260925 00:55:56.827075 31475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:55407
W20260925 00:55:56.928514 31804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:56.928844 31804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:56.928911 31804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:56.933072 31804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:56.933147 31804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:56.933231 31804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:55:56.938741 31804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=44647
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:56.939878 31804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:56.940953 31804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:56.950350 31809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.950340 31812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:56.950340 31810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:56.950639 31804 server_base.cc:1034] running on GCE node
I20260925 00:55:56.951138 31804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:56.951834 31804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:56.953052 31804 hybrid_clock.cc:648] HybridClock initialized: now 1790297756953035 us; error 29 us; skew 500 ppm
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:56.955677 31804 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:55:56.956568 31804 webserver.cc:460] Webserver started at http://127.16.202.3:44647/ using document root <none> and password file <none>
I20260925 00:55:56.957139 31804 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:56.957222 31804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:56.961370 31804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:56.963711 31819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.965112 31804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:56.965219 31804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "cc74147cae58406a9be23feace042daa"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3687cde9c2b6c224740ea7a5af252542"
server_key_iv: "7b31e812ca5018e224e465f9af83a7aa"
server_key_version: "encryptionkey@0"
I20260925 00:55:56.965806 31804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:56.978242 31804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:56.981374 31804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:56.981554 31804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:56.982151 31804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:56.982980 31804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:56.983026 31804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.983069 31804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:56.983187 31804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:56.992614 31804 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:40813
I20260925 00:55:56.992633 31932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:40813 every 8 connection(s)
I20260925 00:55:56.993710 31804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:55:56.998642 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31804
Sep 25 00:55:56 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:55:57.007673 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.999257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.16.202.62:39987)
0925 00:55:56.999458 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.999461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.999616 (+ 155us) server_negotiation.cc:407] Connection header received
0925 00:55:56.999728 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.999731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.999772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.999845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:57.000555 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.001104 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.002517 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.002742 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.002829 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.004301 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:57.004304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:57.004305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:57.004329 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:57.006046 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:57.006426 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:57.006430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:57.006432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:57.006517 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:57.006779 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:57.006783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:57.006784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:57.006986 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:57.007168 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.007360 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.007490 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260925 00:55:57.008085 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.995564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:55693 (local address 127.16.202.62:39987)
0925 00:55:56.995728 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:56.995732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:56.996608 (+ 876us) server_negotiation.cc:407] Connection header received
0925 00:55:56.997934 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:56.997937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:56.997983 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:56.998055 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:56.999449 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.000002 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.001197 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.001426 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.001502 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.004285 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:57.004289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:57.004292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:57.004317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:57.005901 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:57.006448 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:57.006451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:57.006452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:57.006546 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:57.006906 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:57.006909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:57.006911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:57.007123 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:57.007225 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.007842 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.007953 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":4992}
I20260925 00:55:57.009227 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:56.995945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:55693)
0925 00:55:56.996456 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:56.996500 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:56.997751 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:56.998215 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:56.998222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:56.998624 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:56.999246 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:56.999275 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.000129 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.000132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.001089 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.001098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.001619 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.001622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.001774 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.002457 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:57.002475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:57.004104 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:57.006031 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:57.006038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:57.006057 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:57.006355 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:57.006634 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:57.006637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:57.006638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:57.006792 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:57.007236 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:57.007242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:57.007568 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 00:55:57.007777 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 00:55:57.008143 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260925 00:55:57.010339 31933 heartbeater.cc:347] Connected to a master server at 127.16.202.62:39987
I20260925 00:55:57.010592 31933 heartbeater.cc:464] Registering TS with master...
I20260925 00:55:57.011085 31933 heartbeater.cc:511] Master 127.16.202.62:39987 requested a full tablet report, sending...
I20260925 00:55:57.012082 31475 ts_manager.cc:194] Registered new tserver with Master: cc74147cae58406a9be23feace042daa (127.16.202.3:40813)
I20260925 00:55:57.012604 31475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:55693
I20260925 00:55:57.021708 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:55:57.026546 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.022413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.16.202.62:39987)
0925 00:55:57.022547 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:57.022551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:57.022802 (+ 251us) server_negotiation.cc:407] Connection header received
0925 00:55:57.022964 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:57.022967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:57.023087 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:57.023185 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:55:57.023521 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.024053 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.025114 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.025343 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.025436 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.025779 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:55:57.026216 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:55:57.026307 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.026335 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.026384 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:55:57.027222 31475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:55:57.030555 31475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:55:57.038939 31597 tablet_service.cc:1467] Processing CreateTablet for tablet 80133cb444694ee1b0314cbcf806c584 (DEFAULT_TABLE table=test-table [id=81bee03e84f040beac8a73b23c290487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:57.039647 31597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80133cb444694ee1b0314cbcf806c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:57.042554 31555 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Bootstrap starting.
I20260925 00:55:57.043077 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.037915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37639 (local address 127.0.0.1:51220)
0925 00:55:57.038184 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.038196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:57.038273 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:57.039189 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:57.039192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:57.039211 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:57.039325 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.039332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.040829 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.040832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.041757 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.041767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.042648 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.042652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.042738 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.042787 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 00:55:57.042838 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 00:55:57.042924 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
I20260925 00:55:57.043423 31555 tablet_bootstrap.cc:654] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:57.043884 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.038349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.16.202.2:37639)
0925 00:55:57.038664 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:57.038674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:57.038692 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:55:57.038767 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:57.038773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:57.039007 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:57.039177 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:57.039447 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.040662 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.041897 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.042516 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.043418 (+ 902us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.043577 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.043633 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.043731 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":145,"threads_started":1}
I20260925 00:55:57.044299 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.037702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:40813 (local address 127.0.0.1:36694)
0925 00:55:57.037944 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.037960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:57.038033 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:57.039370 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:57.039372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:57.039389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:57.039479 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.039486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.041147 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.041150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.042267 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.042279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.043888 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.043892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.044015 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.044042 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:55:57.044118 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:55:57.044200 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 00:55:57.044713 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.037905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.16.202.3:40813)
0925 00:55:57.038644 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:57.038656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:57.038683 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:55:57.038768 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:57.038773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:57.039007 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:57.039177 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:57.039610 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.040995 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.042418 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.043783 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.043891 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.044033 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.044473 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.044577 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":122,"threads_started":1}
I20260925 00:55:57.045269 31555 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: No bootstrap required, opened a new log
I20260925 00:55:57.045356 31555 ts_tablet_manager.cc:1397] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 00:55:57.045786 31555 raft_consensus.cc:348] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.045914 31555 raft_consensus.cc:374] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:57.045950 31555 raft_consensus.cc:729] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35258f82d3a140e59918df23289f3bd6, State: Initialized, Role: FOLLOWER
I20260925 00:55:57.046075 31555 consensus_queue.cc:260] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.046327 31555 ts_tablet_manager.cc:1428] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 00:55:57.047168 31735 tablet_service.cc:1467] Processing CreateTablet for tablet 80133cb444694ee1b0314cbcf806c584 (DEFAULT_TABLE table=test-table [id=81bee03e84f040beac8a73b23c290487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:57.047688 31867 tablet_service.cc:1467] Processing CreateTablet for tablet 80133cb444694ee1b0314cbcf806c584 (DEFAULT_TABLE table=test-table [id=81bee03e84f040beac8a73b23c290487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:55:57.048237 31735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80133cb444694ee1b0314cbcf806c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:57.048774 31867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80133cb444694ee1b0314cbcf806c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:55:57.055732 31946 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Bootstrap starting.
I20260925 00:55:57.056538 31947 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Bootstrap starting.
I20260925 00:55:57.058104 31946 tablet_bootstrap.cc:654] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:57.058707 31947 tablet_bootstrap.cc:654] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Neither blocks nor log segments found. Creating new log.
I20260925 00:55:57.058964 31946 log.cc:824] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:57.059386 31947 log.cc:824] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:57.060856 31946 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: No bootstrap required, opened a new log
I20260925 00:55:57.061039 31946 ts_tablet_manager.cc:1397] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:55:57.061134 31947 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: No bootstrap required, opened a new log
I20260925 00:55:57.061311 31947 ts_tablet_manager.cc:1397] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:55:57.064039 31946 raft_consensus.cc:348] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.064036 31947 raft_consensus.cc:348] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.064289 31946 raft_consensus.cc:374] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:57.064291 31947 raft_consensus.cc:374] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:55:57.064345 31947 raft_consensus.cc:729] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc74147cae58406a9be23feace042daa, State: Initialized, Role: FOLLOWER
I20260925 00:55:57.064361 31946 raft_consensus.cc:729] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7984fab6cdc4dfcab6dad02799a1cda, State: Initialized, Role: FOLLOWER
I20260925 00:55:57.064841 31946 consensus_queue.cc:260] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.064841 31947 consensus_queue.cc:260] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.065523 31801 heartbeater.cc:503] Master 127.16.202.62:39987 was elected leader, sending a full tablet report...
I20260925 00:55:57.065708 31933 heartbeater.cc:503] Master 127.16.202.62:39987 was elected leader, sending a full tablet report...
I20260925 00:55:57.065819 31946 ts_tablet_manager.cc:1428] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:55:57.065819 31947 ts_tablet_manager.cc:1428] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260925 00:55:57.119390 31664 tablet.cc:2369] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:55:57.244798 31934 tablet.cc:2369] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:55:57.309055 31802 tablet.cc:2369] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:55:57.311817 31951 raft_consensus.cc:482] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:55:57.312039 31951 raft_consensus.cc:504] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.313241 31951 leader_election.cc:288] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639), 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485)
I20260925 00:55:57.316982 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.313624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:49727 (local address 127.16.202.2:37639)
0925 00:55:57.313759 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:57.313763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:57.314005 (+ 242us) server_negotiation.cc:407] Connection header received
0925 00:55:57.314141 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:57.314143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:57.314186 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:57.314246 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:57.314495 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.314968 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.316046 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.316432 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.316551 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.316606 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.316783 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.316856 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:55:57.317013 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.313495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37639 (local address 127.16.202.3:49727)
0925 00:55:57.313899 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.313910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:57.314028 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:57.314270 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:57.314273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:57.314288 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:57.314395 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.314400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.315073 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.315077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.315920 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.315930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.316551 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.316555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.316649 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.316696 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 00:55:57.316759 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:55:57.316830 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":22,"thread_start_us":96,"threads_started":1}
I20260925 00:55:57.317034 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.313495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46485 (local address 127.16.202.3:48841)
0925 00:55:57.313710 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.313725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:57.313822 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:57.314118 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:57.314120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:57.314136 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:55:57.314242 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.314247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.315057 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.315061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.315930 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:57.315939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.316607 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.316611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:57.316737 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.316757 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:55:57.316820 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:55:57.316888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":21}
I20260925 00:55:57.317037 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.313603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:48841 (local address 127.16.202.1:46485)
0925 00:55:57.313874 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:57.313878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:57.313891 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:55:57.313942 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:57.313944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:57.313994 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:57.314089 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:55:57.314341 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.314853 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.316052 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.316495 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.316607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.316799 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.316839 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.316924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260925 00:55:57.317626 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80133cb444694ee1b0314cbcf806c584" candidate_uuid: "cc74147cae58406a9be23feace042daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35258f82d3a140e59918df23289f3bd6" is_pre_election: true
I20260925 00:55:57.317637 31754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80133cb444694ee1b0314cbcf806c584" candidate_uuid: "cc74147cae58406a9be23feace042daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" is_pre_election: true
I20260925 00:55:57.317934 31754 raft_consensus.cc:2454] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc74147cae58406a9be23feace042daa in term 0.
I20260925 00:55:57.317922 31608 raft_consensus.cc:2454] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc74147cae58406a9be23feace042daa in term 0.
I20260925 00:55:57.318374 31823 leader_election.cc:302] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc74147cae58406a9be23feace042daa, f7984fab6cdc4dfcab6dad02799a1cda; no voters:
I20260925 00:55:57.318634 31951 raft_consensus.cc:2781] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:55:57.318704 31951 raft_consensus.cc:482] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:55:57.318734 31951 raft_consensus.cc:3037] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:57.319741 31951 raft_consensus.cc:504] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.320077 31951 leader_election.cc:288] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [CANDIDATE]: Term 1 election: Requested vote from peers f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639), 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485)
I20260925 00:55:57.320453 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80133cb444694ee1b0314cbcf806c584" candidate_uuid: "cc74147cae58406a9be23feace042daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35258f82d3a140e59918df23289f3bd6"
I20260925 00:55:57.320504 31754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80133cb444694ee1b0314cbcf806c584" candidate_uuid: "cc74147cae58406a9be23feace042daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
I20260925 00:55:57.320600 31608 raft_consensus.cc:3037] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:57.320611 31754 raft_consensus.cc:3037] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:55:57.321885 31754 raft_consensus.cc:2454] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc74147cae58406a9be23feace042daa in term 1.
I20260925 00:55:57.321884 31608 raft_consensus.cc:2454] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc74147cae58406a9be23feace042daa in term 1.
I20260925 00:55:57.322182 31821 leader_election.cc:302] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35258f82d3a140e59918df23289f3bd6, cc74147cae58406a9be23feace042daa; no voters:
I20260925 00:55:57.322389 31951 raft_consensus.cc:2781] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:55:57.322587 31951 raft_consensus.cc:686] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 1 LEADER]: Becoming Leader. State: Replica: cc74147cae58406a9be23feace042daa, State: Running, Role: LEADER
I20260925 00:55:57.322892 31951 consensus_queue.cc:237] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.326256 31473 catalog_manager.cc:5495] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa reported cstate change: term changed from 0 to 1, leader changed from <none> to cc74147cae58406a9be23feace042daa (127.16.202.3). New cstate: current_term: 1 leader_uuid: "cc74147cae58406a9be23feace042daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } health_report { overall_health: UNKNOWN } } }
I20260925 00:55:57.340734 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.337364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.16.202.3:40813)
0925 00:55:57.337489 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:57.337493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:57.337508 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:55:57.337676 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:57.337678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:57.337732 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:57.337818 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:55:57.338040 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.338540 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.339327 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:57.339538 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:57.339621 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:57.339902 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:55:57.340311 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:55:57.340419 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:57.340478 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 00:55:57.340537 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:55:57.346567 31608 raft_consensus.cc:1261] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Refusing update from remote peer cc74147cae58406a9be23feace042daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:57.346580 31754 raft_consensus.cc:1261] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Refusing update from remote peer cc74147cae58406a9be23feace042daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:55:57.347307 31951 consensus_queue.cc:1035] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:57.347525 31954 consensus_queue.cc:1035] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:55:57.362879 31473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:57.363090 31473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:55:57.365633 31473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Sending DeleteTablet for 3 replicas of tablet 80133cb444694ee1b0314cbcf806c584
I20260925 00:55:57.366418 31597 tablet_service.cc:1514] Processing DeleteTablet for tablet 80133cb444694ee1b0314cbcf806c584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:57 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260925 00:55:57.366446 31735 tablet_service.cc:1514] Processing DeleteTablet for tablet 80133cb444694ee1b0314cbcf806c584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:57 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:51220
I20260925 00:55:57.366648 31867 tablet_service.cc:1514] Processing DeleteTablet for tablet 80133cb444694ee1b0314cbcf806c584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:55:57 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36694
I20260925 00:55:57.366976 31968 tablet_replica.cc:330] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: stopping tablet replica
I20260925 00:55:57.367034 31967 tablet_replica.cc:330] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: stopping tablet replica
I20260925 00:55:57.367188 31967 raft_consensus.cc:2229] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:57.367188 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31513
I20260925 00:55:57.368644 31968 raft_consensus.cc:2229] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:55:57.369043 31968 raft_consensus.cc:2258] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:57.369467 31969 tablet_replica.cc:330] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: stopping tablet replica
I20260925 00:55:57.369751 31969 raft_consensus.cc:2229] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:55:57.370066 31969 raft_consensus.cc:2258] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:55:57.371389 31969 ts_tablet_manager.cc:1905] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:55:57.373974 31969 ts_tablet_manager.cc:1918] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:55:57.374065 31969 log.cc:1196] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/80133cb444694ee1b0314cbcf806c584
I20260925 00:55:57.374365 31969 ts_tablet_manager.cc:1939] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Deleting consensus metadata
W20260925 00:55:57.374637 31460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:46485 (error 108)
I20260925 00:55:57.374851 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31667
I20260925 00:55:57.376529 31462 catalog_manager.cc:4841] TS cc74147cae58406a9be23feace042daa (127.16.202.3:40813): tablet 80133cb444694ee1b0314cbcf806c584 (table test-table [id=81bee03e84f040beac8a73b23c290487]) successfully deleted
I20260925 00:55:57.376814 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.376261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46485 (local address 127.0.0.1:45622)
0925 00:55:57.376445 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.376519 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260925 00:55:57.376981 31460 catalog_manager.cc:4584] TS 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80133cb444694ee1b0314cbcf806c584: Network error: Client connection negotiation failed: client connection to 127.16.202.1:46485: connect: Connection refused (error 111)
I20260925 00:55:57.382272 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31804
I20260925 00:55:57.383896 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.382550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37639 (local address 127.0.0.1:51234)
0925 00:55:57.383741 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.383796 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.2:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1110}
I20260925 00:55:57.387547 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 31442
I20260925 00:55:57.393167 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39987
--webserver_interface=127.16.202.62
--webserver_port=44065
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:55:57.502629 31971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:57.502934 31971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:57.502995 31971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:57.507256 31971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:55:57.507341 31971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:57.507369 31971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:57.507391 31971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:55:57.507417 31971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:55:57.513161 31971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:57.514256 31971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:57.515259 31971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:57.523851 31977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:57.523828 31979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:57.524153 31971 server_base.cc:1034] running on GCE node
W20260925 00:55:57.523828 31976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:57.524864 31971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:57.526057 31971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:57.527232 31971 hybrid_clock.cc:648] HybridClock initialized: now 1790297757527184 us; error 64 us; skew 500 ppm
Sep 25 00:55:57 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297757, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:57.530121 31971 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:55:57.531111 31971 webserver.cc:460] Webserver started at http://127.16.202.62:44065/ using document root <none> and password file <none>
I20260925 00:55:57.531680 31971 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:57.531754 31971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:57.535719 31971 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:55:57.537909 31986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:57.539178 31971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:55:57.539294 31971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "87b204759c024529966622aae907aabc"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "843c0f96bcb6c98d812d09a860eff6fa"
server_key_iv: "ee9f91e7eccf0f795bb5af76a5b0fa70"
server_key_version: "encryptionkey@0"
I20260925 00:55:57.539824 31971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:57.558274 31971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:57.561416 31971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:57.561607 31971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:57.568543 31971 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39987
I20260925 00:55:57.568876 32038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39987 every 8 connection(s)
I20260925 00:55:57.569694 31971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:55:57.573966 32039 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:55:57.575534 32039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap starting.
I20260925 00:55:57.579466 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 31971
I20260925 00:55:57.579838 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:46485
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:39987
--webserver_port=39671
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:57.593391 32039 log.cc:824] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:57.606454 32039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:55:57.606796 32039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: Bootstrap complete.
I20260925 00:55:57.610620 32039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:57.611018 32039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b204759c024529966622aae907aabc, State: Initialized, Role: FOLLOWER
I20260925 00:55:57.611490 32039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:57.611608 32039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:55:57.611678 32039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:55:57.611776 32039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 2 FOLLOWER]: Advancing to term 3
I20260925 00:55:57.613770 32039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:57.614090 32039 leader_election.cc:302] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b204759c024529966622aae907aabc; no voters:
I20260925 00:55:57.614398 32039 leader_election.cc:288] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 00:55:57.614494 32043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 3 FOLLOWER]: Leader election won for term 3
I20260925 00:55:57.614706 32043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [term 3 LEADER]: Becoming Leader. State: Replica: 87b204759c024529966622aae907aabc, State: Running, Role: LEADER
I20260925 00:55:57.615013 32043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } }
I20260925 00:55:57.615705 32039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:55:57.616506 32045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b204759c024529966622aae907aabc. Latest consensus state: current_term: 3 leader_uuid: "87b204759c024529966622aae907aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } } }
I20260925 00:55:57.616631 32045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:57.616895 32044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "87b204759c024529966622aae907aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b204759c024529966622aae907aabc" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39987 } } }
I20260925 00:55:57.616982 32044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc [sys.catalog]: This master's current role is: LEADER
I20260925 00:55:57.620188 32058 catalog_manager.cc:1259] Loaded cluster ID: 5a1813a5dc184d1396efe8f7b58e92ac
I20260925 00:55:57.620239 32058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: loading cluster ID for follower catalog manager: success
W20260925 00:55:57.621102 32058 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 87b204759c024529966622aae907aabc: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260925 00:55:57.621162 32058 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260925 00:55:57.621200 32052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:55:57.623276 32052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:55:57.623677 32052 catalog_manager.cc:1259] Loaded cluster ID: 5a1813a5dc184d1396efe8f7b58e92ac
I20260925 00:55:57.623724 32052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 00:55:57.624258 32052 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f39acb259ad google::LogMessage::Fail() at ??:0
@ 0x7f39acb2888a google::LogMessage::SendToLog() at ??:0
@ 0x7f39acb254de google::LogMessage::Flush() at ??:0
@ 0x7f39acb26a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f39b7464fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f39b7465d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f39b7462579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f39b74a6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f39aed773d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f39aed77c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f39aed793f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f39aed68512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39b03ac6db start_thread at ??:0
@ 0x7f39ab62361f clone at ??:0
W20260925 00:55:57.715106 32041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:57.715427 32041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:57.715497 32041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:57.719607 32041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:57.719664 32041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:57.719743 32041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:55:57.725137 32041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:46485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=39671
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:57.726294 32041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:57.727334 32041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:57.737285 32083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:57.737310 32081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:57.737289 32080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:57.738195 32041 server_base.cc:1034] running on GCE node
I20260925 00:55:57.738662 32041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:57.739380 32041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:57.740613 32041 hybrid_clock.cc:648] HybridClock initialized: now 1790297757740588 us; error 45 us; skew 500 ppm
Sep 25 00:55:57 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297757, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:57.743777 32041 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:55:57.745054 32041 webserver.cc:460] Webserver started at http://127.16.202.1:39671/ using document root <none> and password file <none>
I20260925 00:55:57.745843 32041 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:57.745944 32041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:57.750654 32041 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:55:57.753564 32090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:57.755314 32041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:55:57.755470 32041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "35258f82d3a140e59918df23289f3bd6"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "2d53d5d3640dc1c587fdaba954609e0f"
server_key_iv: "35b278b9fca6ab321cb6eba6fc33ca77"
server_key_version: "encryptionkey@0"
I20260925 00:55:57.756289 32041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:57.786502 32041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:57.791349 32041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:57.791605 32041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:57.792364 32041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:57.793905 32097 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:55:57.799723 32041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:55:57.799822 32041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 00:55:57.799962 32041 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:55:57.801908 32041 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:55:57.801977 32041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 00:55:57.802368 32097 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Bootstrap starting.
I20260925 00:55:57.813825 32041 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:46485
I20260925 00:55:57.815446 32041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:55:57.816260 32204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:46485 every 8 connection(s)
I20260925 00:55:57.817291 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 32041
I20260925 00:55:57.817716 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:37639
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:39987
--webserver_port=34087
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:57.827335 32097 log.cc:824] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:57.836856 32097 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:55:57.837396 32097 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Bootstrap complete.
I20260925 00:55:57.838806 32097 ts_tablet_manager.cc:1397] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.004s
I20260925 00:55:57.842247 32097 raft_consensus.cc:348] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.842963 32097 raft_consensus.cc:729] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35258f82d3a140e59918df23289f3bd6, State: Initialized, Role: FOLLOWER
I20260925 00:55:57.843585 32097 consensus_queue.cc:260] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:57.844789 32097 ts_tablet_manager.cc:1428] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 00:55:57.935736 32207 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:55:57.817847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:36283)
0925 00:55:57.818400 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.818462 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:57.819573 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:57.934663 (+115090us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
W20260925 00:55:57.936352 32205 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:39987 (0 consecutive failures): Network error: Failed to ping master at 127.16.202.62:39987: Client connection negotiation failed: client connection to 127.16.202.62:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:55:57.937239 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.936888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:52439)
0925 00:55:57.937067 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.937155 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 00:55:57.938059 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:57.937811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:38345)
0925 00:55:57.937936 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:57.937992 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:55:57.938272 32205 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:55:57.944561 32208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:57.944900 32208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:57.944952 32208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:57.949179 32208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:57.949247 32208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:57.949309 32208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:55:57.954896 32208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:37639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=34087
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:57.955981 32208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:57.957067 32208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:57.967190 32216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:57.967219 32218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:57.967182 32215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:57.967449 32208 server_base.cc:1034] running on GCE node
I20260925 00:55:57.967969 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:57.968700 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:57.969877 32208 hybrid_clock.cc:648] HybridClock initialized: now 1790297757969849 us; error 40 us; skew 500 ppm
Sep 25 00:55:57 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297757, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:57.972572 32208 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:55:57.973500 32208 webserver.cc:460] Webserver started at http://127.16.202.2:34087/ using document root <none> and password file <none>
I20260925 00:55:57.974035 32208 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:57.974094 32208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:57.978062 32208 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:55:57.980293 32225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:57.981693 32208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:55:57.981814 32208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3c464da8d5604da55a4100016b90cb3f"
server_key_iv: "3679d40e3cc61ccf8dd09f4461750c0f"
server_key_version: "encryptionkey@0"
I20260925 00:55:57.982390 32208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:58.004648 32208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:58.007889 32208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:58.008111 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:58.008774 32208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:58.009958 32232 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:55:58.014318 32208 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:55:58.014402 32208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 00:55:58.014477 32208 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:55:58.016252 32208 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:55:58.016306 32208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 00:55:58.016635 32232 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Bootstrap starting.
I20260925 00:55:58.026006 32208 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:37639
I20260925 00:55:58.026140 32339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:37639 every 8 connection(s)
I20260925 00:55:58.027053 32208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:55:58.030772 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.029210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:37761)
0925 00:55:58.029778 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.029917 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":157,"threads_started":1}
W20260925 00:55:58.031261 32340 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:39987 (0 consecutive failures): Network error: Failed to ping master at 127.16.202.62:39987: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
I20260925 00:55:58.032174 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.031884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:44509)
0925 00:55:58.032045 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.032101 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 00:55:58.032896 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.032650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:60037)
0925 00:55:58.032797 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.032842 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 00:55:58.033079 32340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:55:58.035145 32232 log.cc:824] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Log is configured to *not* fsync() on all Append() calls
I20260925 00:55:58.036010 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 32208
I20260925 00:55:58.036441 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:40813
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:39987
--webserver_port=44647
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:55:58.043732 32232 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:55:58.044176 32232 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Bootstrap complete.
I20260925 00:55:58.045590 32232 ts_tablet_manager.cc:1397] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.007s
I20260925 00:55:58.049507 32232 raft_consensus.cc:348] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:58.050120 32232 raft_consensus.cc:729] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7984fab6cdc4dfcab6dad02799a1cda, State: Initialized, Role: FOLLOWER
I20260925 00:55:58.050686 32232 consensus_queue.cc:260] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:58.051776 32232 ts_tablet_manager.cc:1428] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 00:55:58.051975 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.051688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:33861)
0925 00:55:58.051821 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.051886 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 00:55:58.052695 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.052445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:52735)
0925 00:55:58.052550 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.052599 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 00:55:58.156224 32344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:55:58.156527 32344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:55:58.156574 32344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:55:58.160794 32344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:55:58.160876 32344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:55:58.160945 32344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:55:58.166608 32344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=44647
--tserver_master_addrs=127.16.202.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:55:58.167704 32344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:55:58.168749 32344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:55:58.178272 32350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:58.178272 32351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:55:58.178399 32353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:55:58.178694 32344 server_base.cc:1034] running on GCE node
I20260925 00:55:58.179131 32344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:55:58.179859 32344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:55:58.181016 32344 hybrid_clock.cc:648] HybridClock initialized: now 1790297758180996 us; error 34 us; skew 500 ppm
Sep 25 00:55:58 dist-test-slave-rpk1 krb5kdc[30894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:55:58.183735 32344 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:55:58.184619 32344 webserver.cc:460] Webserver started at http://127.16.202.3:44647/ using document root <none> and password file <none>
I20260925 00:55:58.185184 32344 fs_manager.cc:359] Metadata directory not provided
I20260925 00:55:58.185243 32344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:55:58.189296 32344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:55:58.191313 32360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:58.192516 32344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:55:58.192612 32344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "cc74147cae58406a9be23feace042daa"
format_stamp: "Formatted at 2026-09-25 00:55:55 on dist-test-slave-rpk1"
server_key: "3687cde9c2b6c224740ea7a5af252542"
server_key_iv: "7b31e812ca5018e224e465f9af83a7aa"
server_key_version: "encryptionkey@0"
I20260925 00:55:58.193174 32344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:55:58.206581 32344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:55:58.209847 32344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:55:58.210031 32344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:55:58.210621 32344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:55:58.211460 32344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:55:58.211521 32344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:58.211607 32344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:55:58.211638 32344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:55:58.220100 32344 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:40813
I20260925 00:55:58.220177 32473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:40813 every 8 connection(s)
I20260925 00:55:58.221168 32344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:55:58.225188 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.223399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:33703)
0925 00:55:58.223942 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.224358 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":106,"threads_started":1}
W20260925 00:55:58.225662 32474 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:39987 (0 consecutive failures): Network error: Failed to ping master at 127.16.202.62:39987: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
I20260925 00:55:58.225992 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 32344
I20260925 00:55:58.226370 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.226122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:56341)
0925 00:55:58.226248 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.226301 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 00:55:58.227025 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.226838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:40363)
0925 00:55:58.226932 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.226974 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 00:55:58.227170 32474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:55:58.939294 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:58.938860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:42183)
0925 00:55:58.939129 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:58.939187 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20260925 00:55:59.053982 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.053458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:35635)
0925 00:55:59.053766 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.053881 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
I20260925 00:55:59.143424 32480 raft_consensus.cc:482] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:55:59.143647 32480 raft_consensus.cc:504] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:59.144829 32480 leader_election.cc:288] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc74147cae58406a9be23feace042daa (127.16.202.3:40813), f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639)
Sep 25 00:55:59 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297757, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.3@KRBTEST.COM
Sep 25 00:55:59 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297757, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.2@KRBTEST.COM
I20260925 00:55:59.162186 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.145303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:37639 (local address 127.16.202.1:43537)
0925 00:55:59.145662 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.145676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:59.145773 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:59.146880 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:59.146885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:59.147420 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:59.148170 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:59.148192 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.150096 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.150100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:59.151607 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:59.151620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.152652 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.152656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:59.152992 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:59.153740 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:59.153763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:59.156242 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:59.158713 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:59.158721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:59.158738 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:59.159271 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:59.159783 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:59.159787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:59.159789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:59.160047 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:59.160950 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:59.160958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:59.161335 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 00:55:59.161580 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 00:55:59.161978 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":400,"spinlock_wait_cycles":53632,"thread_start_us":135,"threads_started":1}
I20260925 00:55:59.162263 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.145184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:57701 (local address 127.16.202.3:40813)
0925 00:55:59.145548 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:59.145565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:59.145615 (+ 50us) server_negotiation.cc:407] Connection header received
0925 00:55:59.146514 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:59.146522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:59.146905 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:59.147178 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:59.148368 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.149934 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.151743 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.152544 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.152719 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:59.156268 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:59.156274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:59.156289 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:59.156325 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:59.158550 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:59.159453 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:59.159458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:59.159463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:59.159652 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:59.160162 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:59.160166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:59.160168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:59.160801 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:59.160949 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:59.161752 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 00:55:59.162073 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":65,"threads_started":1}
I20260925 00:55:59.162246 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.145467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:43537 (local address 127.16.202.2:37639)
0925 00:55:59.145750 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 00:55:59.145762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:55:59.145795 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:55:59.145973 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:55:59.145979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:55:59.146430 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:55:59.146763 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:55:59.148366 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.149934 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.151743 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.152501 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.152715 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:59.156422 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:55:59.156429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:55:59.156445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:55:59.156479 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:55:59.158550 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:59.159439 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:59.159446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:59.159453 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:59.159652 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:55:59.160162 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:55:59.160166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:55:59.160168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:55:59.160801 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:55:59.160949 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 00:55:59.161772 (+ 823us) server_negotiation.cc:299] Negotiation successful
0925 00:55:59.162073 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260925 00:55:59.162215 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.145025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:40813 (local address 127.16.202.1:57701)
0925 00:55:59.145163 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.145178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:55:59.145278 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:55:59.147139 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:55:59.147142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:55:59.147420 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:55:59.148170 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:59.148192 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.150096 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.150100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:59.151610 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:55:59.151620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:55:59.152652 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:55:59.152656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:55:59.153010 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:55:59.153736 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:55:59.153773 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:55:59.156018 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:55:59.158713 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:59.158721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:59.158738 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:59.159271 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:59.159783 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:55:59.159787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:55:59.159789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:55:59.160047 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:55:59.160950 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:55:59.160958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:55:59.161335 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 00:55:59.161580 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 00:55:59.162081 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":425,"spinlock_wait_cycles":135680}
I20260925 00:55:59.163760 32294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80133cb444694ee1b0314cbcf806c584" candidate_uuid: "35258f82d3a140e59918df23289f3bd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" is_pre_election: true
I20260925 00:55:59.164115 32294 raft_consensus.cc:2454] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35258f82d3a140e59918df23289f3bd6 in term 1.
I20260925 00:55:59.165350 32094 leader_election.cc:302] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35258f82d3a140e59918df23289f3bd6, f7984fab6cdc4dfcab6dad02799a1cda; no voters:
I20260925 00:55:59.165627 32480 raft_consensus.cc:2781] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 00:55:59.165733 32480 raft_consensus.cc:482] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 00:55:59.165721 32094 leader_election.cc:341] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc74147cae58406a9be23feace042daa (127.16.202.3:40813): Not found: Tablet not found: 80133cb444694ee1b0314cbcf806c584
I20260925 00:55:59.165853 32480 raft_consensus.cc:3037] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:55:59.168224 32480 raft_consensus.cc:504] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:59.168597 32480 leader_election.cc:288] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 2 election: Requested vote from peers cc74147cae58406a9be23feace042daa (127.16.202.3:40813), f7984fab6cdc4dfcab6dad02799a1cda (127.16.202.2:37639)
I20260925 00:55:59.169234 32294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80133cb444694ee1b0314cbcf806c584" candidate_uuid: "35258f82d3a140e59918df23289f3bd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f7984fab6cdc4dfcab6dad02799a1cda"
I20260925 00:55:59.169415 32294 raft_consensus.cc:3037] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 1 FOLLOWER]: Advancing to term 2
W20260925 00:55:59.169512 32094 leader_election.cc:341] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer cc74147cae58406a9be23feace042daa (127.16.202.3:40813): Not found: Tablet not found: 80133cb444694ee1b0314cbcf806c584
I20260925 00:55:59.171852 32294 raft_consensus.cc:2454] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35258f82d3a140e59918df23289f3bd6 in term 2.
I20260925 00:55:59.172246 32094 leader_election.cc:302] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35258f82d3a140e59918df23289f3bd6, f7984fab6cdc4dfcab6dad02799a1cda; no voters: cc74147cae58406a9be23feace042daa
I20260925 00:55:59.172452 32480 raft_consensus.cc:2781] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:55:59.172645 32480 raft_consensus.cc:686] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [term 2 LEADER]: Becoming Leader. State: Replica: 35258f82d3a140e59918df23289f3bd6, State: Running, Role: LEADER
I20260925 00:55:59.173034 32480 consensus_queue.cc:237] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:55:59.173410 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.173163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:36941)
0925 00:55:59.173285 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.173341 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 00:55:59.228235 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.227798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:46777)
0925 00:55:59.228063 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.228125 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
W20260925 00:55:59.615705 32094 consensus_peers.cc:487] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 -> Peer cc74147cae58406a9be23feace042daa (127.16.202.3:40813): Couldn't send request to peer cc74147cae58406a9be23feace042daa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80133cb444694ee1b0314cbcf806c584. This is attempt 1: this message will repeat every 5th retry.
I20260925 00:55:59.664606 32294 raft_consensus.cc:1261] T 80133cb444694ee1b0314cbcf806c584 P f7984fab6cdc4dfcab6dad02799a1cda [term 2 FOLLOWER]: Refusing update from remote peer 35258f82d3a140e59918df23289f3bd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 00:55:59.665416 32480 consensus_queue.cc:1035] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 00:55:59.667624 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.667192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:54833)
0925 00:55:59.667457 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.667521 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":58,"threads_started":1}
I20260925 00:55:59.669785 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.669492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:44847)
0925 00:55:59.669646 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.669702 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 00:55:59.670033 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:55:59.669822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:34635)
0925 00:55:59.669915 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 00:55:59.669973 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 00:56:00.163450 32492 ts_tablet_manager.cc:927] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Initiating tablet copy from peer 35258f82d3a140e59918df23289f3bd6 (127.16.202.1:46485)
I20260925 00:56:00.164019 32492 tablet_copy_client.cc:316] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: tablet copy: Beginning tablet copy session from remote peer at address 127.16.202.1:46485
Sep 25 00:56:00 dist-test-slave-rpk1 krb5kdc[30894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:56:00.174126 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.164506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:46485 (local address 127.16.202.3:39899)
0925 00:56:00.164792 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:00.164813 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:00.164937 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:00.165585 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:00.165591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:00.165969 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:00.166111 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:00.166118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:00.167086 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:00.167089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:00.167786 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:00.167802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:00.168336 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:00.168339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:00.168421 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:00.168995 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:00.169007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:00.170530 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:00.172269 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:00.172278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:00.172283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:00.172613 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:00.172931 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:00.172934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:00.172936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:00.173038 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:00.173495 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:00.173502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:00.173633 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 00:56:00.173756 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 00:56:00.173936 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260925 00:56:00.174165 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.164709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:39899 (local address 127.16.202.1:46485)
0925 00:56:00.165012 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:00.165023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:00.165134 (+ 111us) server_negotiation.cc:407] Connection header received
0925 00:56:00.165201 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:00.165207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:00.165398 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:00.165522 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:00.166253 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:00.166984 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:00.167892 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:00.168247 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:00.168326 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:00.170663 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:00.170672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:00.170683 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:00.170708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:00.172137 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:00.172746 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:00.172751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:00.172753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:00.172836 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:00.173124 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:00.173127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:00.173129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:00.173380 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:00.173551 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:00.173872 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 00:56:00.174017 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260925 00:56:00.175197 32179 tablet_copy_service.cc:140] P 35258f82d3a140e59918df23289f3bd6: Received BeginTabletCopySession request for tablet 80133cb444694ee1b0314cbcf806c584 from peer cc74147cae58406a9be23feace042daa ({username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:39899)
I20260925 00:56:00.175359 32179 tablet_copy_service.cc:161] P 35258f82d3a140e59918df23289f3bd6: Beginning new tablet copy session on tablet 80133cb444694ee1b0314cbcf806c584 from peer cc74147cae58406a9be23feace042daa at {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:39899: session id = cc74147cae58406a9be23feace042daa-80133cb444694ee1b0314cbcf806c584
I20260925 00:56:00.176604 32179 tablet_copy_source_session.cc:205] T 80133cb444694ee1b0314cbcf806c584 P 35258f82d3a140e59918df23289f3bd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 00:56:00.180612 32492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80133cb444694ee1b0314cbcf806c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:00.187357 32492 tablet_copy_client.cc:791] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: tablet copy: Starting download of 0 data blocks...
I20260925 00:56:00.187554 32492 tablet_copy_client.cc:655] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: tablet copy: Starting download of 1 WAL segments...
I20260925 00:56:00.189366 32492 tablet_copy_client.cc:531] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 00:56:00.191192 32492 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Bootstrap starting.
I20260925 00:56:00.196925 32492 log.cc:824] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:00.203625 32492 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:56:00.203933 32492 tablet_bootstrap.cc:492] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Bootstrap complete.
I20260925 00:56:00.204252 32492 ts_tablet_manager.cc:1397] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260925 00:56:00.207491 32492 raft_consensus.cc:348] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:56:00.207778 32492 raft_consensus.cc:729] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc74147cae58406a9be23feace042daa, State: Initialized, Role: FOLLOWER
I20260925 00:56:00.208222 32492 consensus_queue.cc:260] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc74147cae58406a9be23feace042daa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 40813 } } peers { permanent_uuid: "f7984fab6cdc4dfcab6dad02799a1cda" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 37639 } } peers { permanent_uuid: "35258f82d3a140e59918df23289f3bd6" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 46485 } }
I20260925 00:56:00.209265 32492 ts_tablet_manager.cc:1428] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260925 00:56:00.209510 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.209241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:36807)
0925 00:56:00.209374 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:00.209435 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:56:00.209929 32179 tablet_copy_service.cc:342] P 35258f82d3a140e59918df23289f3bd6: Request end of tablet copy session cc74147cae58406a9be23feace042daa-80133cb444694ee1b0314cbcf806c584 received from {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:39899
I20260925 00:56:00.210032 32179 tablet_copy_service.cc:433] P 35258f82d3a140e59918df23289f3bd6: ending tablet copy session cc74147cae58406a9be23feace042daa-80133cb444694ee1b0314cbcf806c584 on tablet 80133cb444694ee1b0314cbcf806c584 with peer cc74147cae58406a9be23feace042daa
I20260925 00:56:00.210216 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.209987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:60831)
0925 00:56:00.210105 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:00.210154 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 00:56:00.571118 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.570736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:42593)
0925 00:56:00.570951 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:00.571009 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260925 00:56:00.571277 32428 raft_consensus.cc:1203] T 80133cb444694ee1b0314cbcf806c584 P cc74147cae58406a9be23feace042daa [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 00:56:00.572188 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.571943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:47173)
0925 00:56:00.572062 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:00.572114 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 00:56:00.671563 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:00.671123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:40609)
0925 00:56:00.671380 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:00.671442 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260925 00:56:01.572315 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:01.571903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:52243)
0925 00:56:01.572159 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:01.572218 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260925 00:56:01.573181 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:01.572829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:37391)
0925 00:56:01.573048 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:01.573100 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":60,"threads_started":1}
I20260925 00:56:01.673163 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:01.672484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:59719)
0925 00:56:01.672931 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:01.673019 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":84,"threads_started":1}
I20260925 00:56:02.573663 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:02.573165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:58649)
0925 00:56:02.573471 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:02.573540 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":70,"threads_started":1}
I20260925 00:56:02.574339 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:02.573888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:50185)
0925 00:56:02.574205 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:02.574261 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":71,"threads_started":1}
I20260925 00:56:02.674546 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:02.674067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:47681)
0925 00:56:02.674356 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:02.674421 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260925 00:56:03.575081 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:03.574643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:43151)
0925 00:56:03.574924 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:03.574982 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260925 00:56:03.575663 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:03.575214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:53835)
0925 00:56:03.575539 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:03.575588 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":83,"threads_started":1}
I20260925 00:56:03.575809 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:03.575588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:39895)
0925 00:56:03.575696 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:03.575743 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 00:56:03.676069 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:03.675611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:56395)
0925 00:56:03.675896 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:03.675964 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260925 00:56:04.576923 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:04.576454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:33561)
0925 00:56:04.576728 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:04.576805 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260925 00:56:04.576941 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:04.576453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:45925)
0925 00:56:04.576815 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:04.576866 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":79,"threads_started":1}
I20260925 00:56:04.677464 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:04.676969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:45233)
0925 00:56:04.677297 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:04.677360 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260925 00:56:05.578222 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:05.577806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:51689)
0925 00:56:05.578063 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:05.578127 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260925 00:56:05.578267 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:05.577805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:35725)
0925 00:56:05.578133 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:05.578182 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":80,"threads_started":1}
I20260925 00:56:05.678761 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:05.678255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:44741)
0925 00:56:05.678571 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:05.678638 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
I20260925 00:56:06.579429 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:06.578992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:51991)
0925 00:56:06.579267 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:06.579329 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260925 00:56:06.579483 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:06.578992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:33835)
0925 00:56:06.579345 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:06.579396 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":97,"threads_started":1}
I20260925 00:56:06.680079 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:06.679617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:40613)
0925 00:56:06.679904 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:06.679971 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260925 00:56:07.580745 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:07.580302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:58935)
0925 00:56:07.580567 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:07.580617 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260925 00:56:07.580742 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:07.580302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:50139)
0925 00:56:07.580556 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:07.580614 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260925 00:56:07.681300 32522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:07.680904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:39229)
0925 00:56:07.681146 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:07.681202 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260925 00:56:08.582160 32523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:08.581689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:33723)
0925 00:56:08.581994 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:08.582064 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260925 00:56:08.582187 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:08.581689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:48689)
0925 00:56:08.582045 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:08.582100 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":88,"threads_started":1}
I20260925 00:56:08.682695 32525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:08.682199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:52775)
0925 00:56:08.682518 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:08.682591 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
I20260925 00:56:09.583464 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:09.583044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:52913)
0925 00:56:09.583303 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:09.583365 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260925 00:56:09.583514 32527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:09.583102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:58143)
0925 00:56:09.583367 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:09.583425 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260925 00:56:09.683930 32528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:09.683475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:50201)
0925 00:56:09.683773 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:09.683835 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260925 00:56:10.584714 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:10.584309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:33689)
0925 00:56:10.584542 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:10.584593 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260925 00:56:10.584715 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:10.584267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:47747)
0925 00:56:10.584532 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:10.584585 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260925 00:56:10.685226 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:10.684736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:39953)
0925 00:56:10.685028 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:10.685103 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260925 00:56:11.586021 32534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:11.585542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:58421)
0925 00:56:11.585838 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:11.585909 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":99,"threads_started":1}
I20260925 00:56:11.586021 32533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:11.585542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:60861)
0925 00:56:11.585838 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:11.585920 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260925 00:56:11.686532 32535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:11.686081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:51773)
0925 00:56:11.686365 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:11.686433 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260925 00:56:12.587348 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:12.586873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.3:40951)
0925 00:56:12.587175 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:12.587249 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260925 00:56:12.587369 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:12.586949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.1:47437)
0925 00:56:12.587224 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:12.587281 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260925 00:56:12.687777 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:12.687303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39987 (local address 127.16.202.2:39737)
0925 00:56:12.687618 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:12.687679 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
I20260925 00:56:13.236100 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 32041
I20260925 00:56:13.241619 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 32208
I20260925 00:56:13.246466 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 32344
2026-09-25T00:56:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19300 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:56:13 dist-test-slave-rpk1 krb5kdc[32541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:56:13 dist-test-slave-rpk1 krb5kdc[32541](info): set up 2 sockets
Sep 25 00:56:13 dist-test-slave-rpk1 krb5kdc[32541](info): commencing operation
krb5kdc: starting...
W20260925 00:56:14.224773 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:56:14 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:56:14.348373 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36881
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:56:14.459299 32554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:14.459584 32554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:14.459625 32554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:14.463691 32554 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 00:56:14.463752 32554 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 00:56:14.463766 32554 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 00:56:14.463778 32554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:56:14.463795 32554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:14.463811 32554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:14.463825 32554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:56:14.463842 32554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:56:14.469269 32554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:14.470322 32554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:14.471304 32554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:56:14.480329 32554 server_base.cc:1034] running on GCE node
W20260925 00:56:14.480269 32559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:14.480269 32560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:14.480269 32562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:14.481216 32554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:14.482398 32554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:14.483604 32554 hybrid_clock.cc:648] HybridClock initialized: now 1790297774483575 us; error 38 us; skew 500 ppm
Sep 25 00:56:14 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:14.486486 32554 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:56:14.487416 32554 webserver.cc:460] Webserver started at http://127.16.202.62:37269/ using document root <none> and password file <none>
I20260925 00:56:14.487985 32554 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:14.488049 32554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:14.488216 32554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:56:14.490365 32554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "14953451e860409b980090cb833f7899"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "c7fa245aa6b6558d01bd0d8a29744f80"
server_key_iv: "920f09128ad04e393d7f947f00b71ffc"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.490935 32554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "14953451e860409b980090cb833f7899"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "c7fa245aa6b6558d01bd0d8a29744f80"
server_key_iv: "920f09128ad04e393d7f947f00b71ffc"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.494889 32554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:56:14.497555 32569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:14.499128 32554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:14.499266 32554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "14953451e860409b980090cb833f7899"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "c7fa245aa6b6558d01bd0d8a29744f80"
server_key_iv: "920f09128ad04e393d7f947f00b71ffc"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.499408 32554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:14.533156 32554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:14.536641 32554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:14.536871 32554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:14.543624 32554 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36881
I20260925 00:56:14.543888 32621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36881 every 8 connection(s)
I20260925 00:56:14.544612 32554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:56:14.545303 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 32554
I20260925 00:56:14.545401 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:56:14.545660 17192 external_mini_cluster.cc:1428] Setting key edd00e708c9c7fa72b9727a0035e65aa
I20260925 00:56:14.547894 32622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:14.554792 32622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap starting.
Sep 25 00:56:14 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:14.557328 32622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:14.558058 32622 log.cc:824] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:14.559975 32622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: No bootstrap required, opened a new log
I20260925 00:56:14.560534 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:14.547190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.16.202.62:36881)
0925 00:56:14.547599 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:14.547615 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:14.547649 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:56:14.548311 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:14.548330 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:14.548708 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:14.549004 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:14.549966 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:14.551074 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:14.552230 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:14.552843 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:14.552978 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:14.555866 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:14.555871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:14.555886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:14.555920 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:14.557667 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:14.558112 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:14.558117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:14.558122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:14.558262 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:14.558539 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:14.558543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:14.558545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:14.558954 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:14.559130 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:14.559449 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 00:56:14.559731 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260925 00:56:14.563514 32622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:14.563740 32622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:14.563796 32622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14953451e860409b980090cb833f7899, State: Initialized, Role: FOLLOWER
I20260925 00:56:14.564198 32622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:14.564325 32622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:56:14.564390 32622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:56:14.564492 32622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:14.565439 32622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:14.565766 32622 leader_election.cc:302] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14953451e860409b980090cb833f7899; no voters:
I20260925 00:56:14.566069 32622 leader_election.cc:288] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:56:14.566206 32627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:56:14.566442 32627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 LEADER]: Becoming Leader. State: Replica: 14953451e860409b980090cb833f7899, State: Running, Role: LEADER
I20260925 00:56:14.566742 32627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:14.567261 32622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:56:14.568603 32629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14953451e860409b980090cb833f7899. Latest consensus state: current_term: 1 leader_uuid: "14953451e860409b980090cb833f7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } } }
I20260925 00:56:14.568584 32628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14953451e860409b980090cb833f7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } } }
I20260925 00:56:14.568791 32628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: This master's current role is: LEADER
I20260925 00:56:14.569098 32629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: This master's current role is: LEADER
I20260925 00:56:14.569240 32638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:56:14.572605 32638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:56:14.578747 32638 catalog_manager.cc:1347] Generated new cluster ID: 7923c4d31e8b42c68c87003181cca5a6
I20260925 00:56:14.578850 32638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:56:14.590822 32638 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:56:14.591799 32638 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:56:14.597167 32638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Generated new TSK 0
I20260925 00:56:14.597867 32638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:56:14.632409 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36881
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:56:14.746207 32648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:14.746497 32648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:14.746557 32648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:14.750710 32648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:14.750787 32648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:14.750874 32648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:56:14.756408 32648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:14.757579 32648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:14.758675 32648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:14.768378 32653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:14.768385 32654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:14.768385 32656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:14.768711 32648 server_base.cc:1034] running on GCE node
I20260925 00:56:14.769268 32648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:14.769945 32648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:14.771130 32648 hybrid_clock.cc:648] HybridClock initialized: now 1790297774771108 us; error 35 us; skew 500 ppm
Sep 25 00:56:14 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:14.774148 32648 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:56:14.775113 32648 webserver.cc:460] Webserver started at http://127.16.202.1:38807/ using document root <none> and password file <none>
I20260925 00:56:14.775660 32648 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:14.775720 32648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:14.775903 32648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:56:14.778020 32648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "6363f966d1dc41c78977b86385f7e8c9"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "b70bb0ad3a76ebfc4cdf3b535f7700f1"
server_key_iv: "2c31ed484ab9952fba5a4ea43fc767fb"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.778553 32648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "6363f966d1dc41c78977b86385f7e8c9"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "b70bb0ad3a76ebfc4cdf3b535f7700f1"
server_key_iv: "2c31ed484ab9952fba5a4ea43fc767fb"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.782522 32648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:56:14.785142 32663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:14.786442 32648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:14.786553 32648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "6363f966d1dc41c78977b86385f7e8c9"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "b70bb0ad3a76ebfc4cdf3b535f7700f1"
server_key_iv: "2c31ed484ab9952fba5a4ea43fc767fb"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.786679 32648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:14.799175 32648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:14.802245 32648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:14.802428 32648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:14.803119 32648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:14.803995 32648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:14.804044 32648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:14.804087 32648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:14.804116 32648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:14.813304 32648 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:39183
I20260925 00:56:14.813334 308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:39183 every 8 connection(s)
I20260925 00:56:14.814278 32648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:56:14.819533 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 32648
I20260925 00:56:14.819651 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:56:14.819943 17192 external_mini_cluster.cc:1428] Setting key 9d219a87105cc1d666f51179755d2adb
Sep 25 00:56:14 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:14.828419 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:14.816216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:59293 (local address 127.16.202.62:36881)
0925 00:56:14.816350 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:14.816353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:14.817186 (+ 833us) server_negotiation.cc:407] Connection header received
0925 00:56:14.818583 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:14.818587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:14.818646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:14.818719 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:14.820072 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:14.820754 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:14.821988 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:14.822229 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:14.822318 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:14.824910 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:14.824914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:14.824917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:14.824945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:14.826455 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:14.826931 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:14.826934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:14.826936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:14.827035 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:14.827320 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:14.827323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:14.827325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:14.827550 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:14.827646 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:14.828111 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 00:56:14.828243 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:56:14.829193 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:14.816537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:59293)
0925 00:56:14.817026 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:14.817073 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:14.818344 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:14.818924 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:14.818932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:14.819306 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:14.819913 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:14.819935 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:14.820873 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:14.820876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:14.821865 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:14.821875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:14.822374 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:14.822378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:14.822546 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:14.823113 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:14.823129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:14.824745 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:14.826579 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:14.826584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:14.826596 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:14.826841 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:14.827115 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:14.827117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:14.827118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:14.827240 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:14.827645 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:14.827650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:14.827881 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 00:56:14.828043 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 00:56:14.828299 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260925 00:56:14.830389 309 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36881
I20260925 00:56:14.830637 309 heartbeater.cc:464] Registering TS with master...
I20260925 00:56:14.831110 309 heartbeater.cc:511] Master 127.16.202.62:36881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:56:14.832705 32586 ts_manager.cc:194] Registered new tserver with Master: 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183)
I20260925 00:56:14.834092 32586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:59293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:56:14.848290 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36881
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:56:14.961010 314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:14.961306 314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:14.961367 314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:14.965705 314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:14.965772 314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:14.965857 314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:56:14.971511 314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:14.972632 314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:14.973728 314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:14.983589 319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:14.983666 314 server_base.cc:1034] running on GCE node
W20260925 00:56:14.983609 320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:14.983611 322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:14.984337 314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:14.985101 314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:14.986275 314 hybrid_clock.cc:648] HybridClock initialized: now 1790297774986220 us; error 69 us; skew 500 ppm
Sep 25 00:56:14 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:14.989069 314 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:56:14.990047 314 webserver.cc:460] Webserver started at http://127.16.202.2:38089/ using document root <none> and password file <none>
I20260925 00:56:14.990625 314 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:14.990690 314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:14.990906 314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:56:14.993063 314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "4b2a3b016faaf7511a8c55fac8897691"
server_key_iv: "e396739908f4da63d9b4cbaa073af770"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.993634 314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "4b2a3b016faaf7511a8c55fac8897691"
server_key_iv: "e396739908f4da63d9b4cbaa073af770"
server_key_version: "encryptionkey@0"
I20260925 00:56:14.997538 314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:56:15.000131 329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:15.001529 314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:15.001646 314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "4b2a3b016faaf7511a8c55fac8897691"
server_key_iv: "e396739908f4da63d9b4cbaa073af770"
server_key_version: "encryptionkey@0"
I20260925 00:56:15.001755 314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:15.021955 314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:15.025151 314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:15.025329 314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:15.025905 314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:15.026708 314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:15.026755 314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:15.026808 314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:15.026836 314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:15.035138 314 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:32827
I20260925 00:56:15.035194 442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:32827 every 8 connection(s)
I20260925 00:56:15.036053 314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:56:15.045313 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 314
I20260925 00:56:15.045418 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:56:15.045730 17192 external_mini_cluster.cc:1428] Setting key 6100112b4580dd7b30a67fd0e2a35cbb
Sep 25 00:56:15 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:15.050943 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.038024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:35863 (local address 127.16.202.62:36881)
0925 00:56:15.038181 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.038185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.039046 (+ 861us) server_negotiation.cc:407] Connection header received
0925 00:56:15.040495 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.040498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.040545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.040623 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:15.041917 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.042522 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.043786 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.044016 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.044108 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.046800 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:15.046804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:15.046807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:15.046831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:15.048590 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:15.049228 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:15.049231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:15.049232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:15.049309 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:15.049786 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:15.049789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:15.049791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:15.049988 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:15.050080 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.050705 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.050815 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 00:56:15.052105 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.038378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:35863)
0925 00:56:15.038862 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.038910 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.040233 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.040821 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.040828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.041191 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:15.041744 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.041757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.042670 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.042673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.043645 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.043656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.044147 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.044150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.044292 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.044990 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:15.045007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:15.046609 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:15.048756 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:15.048764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:15.048780 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:15.049122 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:15.049399 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:15.049402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:15.049404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:15.049655 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:15.050092 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:15.050099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:15.050434 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.050640 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.050991 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260925 00:56:15.053388 443 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36881
I20260925 00:56:15.053625 443 heartbeater.cc:464] Registering TS with master...
I20260925 00:56:15.054122 443 heartbeater.cc:511] Master 127.16.202.62:36881 requested a full tablet report, sending...
I20260925 00:56:15.055233 32586 ts_manager.cc:194] Registered new tserver with Master: dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:56:15.055836 32586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:35863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:56:15.076231 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36881
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:56:15.187665 448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:15.187961 448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:15.188030 448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:15.192314 448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:15.192389 448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:15.192471 448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:56:15.198001 448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:15.199111 448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:15.200134 448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:15.209843 454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:15.209867 453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:15.209838 456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:15.209986 448 server_base.cc:1034] running on GCE node
I20260925 00:56:15.210726 448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:15.211385 448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:15.212589 448 hybrid_clock.cc:648] HybridClock initialized: now 1790297775212565 us; error 34 us; skew 500 ppm
Sep 25 00:56:15 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:15.215467 448 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:56:15.216605 448 webserver.cc:460] Webserver started at http://127.16.202.3:39563/ using document root <none> and password file <none>
I20260925 00:56:15.217213 448 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:15.217284 448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:15.217504 448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:56:15.219666 448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "ae007bbaca2943a2b30a9e51ebbc5eec"
format_stamp: "Formatted at 2026-09-25 00:56:15 on dist-test-slave-rpk1"
server_key: "9673cc3bb90feffc1bde3c35b3805bbe"
server_key_iv: "6535b470be4030f4b1458dad3f4a6df9"
server_key_version: "encryptionkey@0"
I20260925 00:56:15.220227 448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "ae007bbaca2943a2b30a9e51ebbc5eec"
format_stamp: "Formatted at 2026-09-25 00:56:15 on dist-test-slave-rpk1"
server_key: "9673cc3bb90feffc1bde3c35b3805bbe"
server_key_iv: "6535b470be4030f4b1458dad3f4a6df9"
server_key_version: "encryptionkey@0"
I20260925 00:56:15.224093 448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:56:15.226689 463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:15.228109 448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:56:15.228237 448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "ae007bbaca2943a2b30a9e51ebbc5eec"
format_stamp: "Formatted at 2026-09-25 00:56:15 on dist-test-slave-rpk1"
server_key: "9673cc3bb90feffc1bde3c35b3805bbe"
server_key_iv: "6535b470be4030f4b1458dad3f4a6df9"
server_key_version: "encryptionkey@0"
I20260925 00:56:15.228374 448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:15.258790 448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:15.262024 448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:15.262219 448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:15.262879 448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:15.263739 448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:15.263795 448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:15.263850 448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:15.263880 448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:15.273224 448 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:38389
I20260925 00:56:15.273234 576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:38389 every 8 connection(s)
I20260925 00:56:15.274207 448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:56:15.283160 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 448
I20260925 00:56:15.283310 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:56:15.283651 17192 external_mini_cluster.cc:1428] Setting key bc59e6119325c5d631f4161f99aa7194
Sep 25 00:56:15 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:15.289196 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.276291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:52995 (local address 127.16.202.62:36881)
0925 00:56:15.276544 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.276549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.277470 (+ 921us) server_negotiation.cc:407] Connection header received
0925 00:56:15.278883 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.278886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.278939 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.279030 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:15.280329 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.281066 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.282330 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.282576 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.282663 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.285565 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:15.285568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:15.285570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:15.285596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:15.286939 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:15.287582 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:15.287585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:15.287587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:15.287667 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:15.288025 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:15.288027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:15.288028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:15.288209 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:15.288283 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.288958 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.289071 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 00:56:15.290125 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.276712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:52995)
0925 00:56:15.277265 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.277323 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.278616 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.279201 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.279211 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.279651 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:15.280169 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.280185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.281220 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.281224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.282195 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.282205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.282753 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.282757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.282994 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.283675 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:15.283697 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:15.285391 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:15.287074 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:15.287080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:15.287098 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:15.287475 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:15.287774 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:15.287777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:15.287778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:15.287940 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:15.288289 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:15.288295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:15.288650 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.288895 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.289189 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":159,"threads_started":1}
I20260925 00:56:15.291410 577 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36881
I20260925 00:56:15.291661 577 heartbeater.cc:464] Registering TS with master...
I20260925 00:56:15.292167 577 heartbeater.cc:511] Master 127.16.202.62:36881 requested a full tablet report, sending...
I20260925 00:56:15.293275 32586 ts_manager.cc:194] Registered new tserver with Master: ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389)
I20260925 00:56:15.293823 32586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:52995
I20260925 00:56:15.297726 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:56:15.307364 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.299803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.16.202.62:36881)
0925 00:56:15.299990 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.299993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.300175 (+ 182us) server_negotiation.cc:407] Connection header received
0925 00:56:15.300245 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.300248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.300292 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.300372 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:15.301185 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.301722 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.302587 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.302892 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.302977 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.304375 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:15.304378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:15.304380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:15.304406 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:15.305713 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:15.306122 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:15.306124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:15.306125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:15.306200 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:15.306522 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:15.306525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:15.306526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:15.306744 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:15.306851 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.307124 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.307231 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 00:56:15.310627 32586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:56:15.313722 32586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:56:15.329192 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.323628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.16.202.3:38389)
0925 00:56:15.324193 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.324203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.324528 (+ 325us) server_negotiation.cc:407] Connection header received
0925 00:56:15.324608 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.324612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.324813 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.324978 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:15.325311 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.326339 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.327606 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.328426 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.328538 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.328724 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.328926 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.329033 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":143,"threads_started":1}
I20260925 00:56:15.329206 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.323145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:38389 (local address 127.0.0.1:52230)
0925 00:56:15.324318 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.324331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.324419 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.324961 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.324964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.324981 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:15.325183 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.325191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.326456 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.326459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.327483 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.327492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.328567 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.328572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.328724 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.328765 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.328889 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.329039 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":44,"thread_start_us":91,"threads_started":1}
I20260925 00:56:15.329274 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.322972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.16.202.2:32827)
0925 00:56:15.323939 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.323951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.323974 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:56:15.324052 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.324058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.324284 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.324535 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:15.324835 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.326001 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.327755 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.328505 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.328590 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.328724 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.329055 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.329143 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":87,"threads_started":1}
I20260925 00:56:15.329535 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.323632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.16.202.1:39183)
0925 00:56:15.324026 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.324039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.324296 (+ 257us) server_negotiation.cc:407] Connection header received
0925 00:56:15.324365 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.324368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.324574 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.324976 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:15.325243 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.326006 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.327901 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.328513 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.328613 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.328786 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.329268 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.329384 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":118,"threads_started":1}
I20260925 00:56:15.329206 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.322961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39183 (local address 127.0.0.1:48546)
0925 00:56:15.324107 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.324120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.324209 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.324734 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.324737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.324754 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:15.324841 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.324847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.326160 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.326162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.327791 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.327800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.328843 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.328847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.328928 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.328948 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.329006 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.329063 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":428,"thread_start_us":111,"threads_started":1}
I20260925 00:56:15.329897 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.322806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32827 (local address 127.0.0.1:34676)
0925 00:56:15.323464 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.323521 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.323650 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.324496 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.324502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.324525 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:15.324721 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.324729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.326143 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.326146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.327634 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.327644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.328789 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.328792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.328864 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.328900 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.328969 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.329767 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":764,"thread_start_us":89,"threads_started":1}
I20260925 00:56:15.332401 32711 tablet_service.cc:1467] Processing CreateTablet for tablet 215b74780cda4c20a79900da4b80525c (DEFAULT_TABLE table=test-table [id=9ddce4892e4149409e95fb7731355f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:56:15.332378 511 tablet_service.cc:1467] Processing CreateTablet for tablet 215b74780cda4c20a79900da4b80525c (DEFAULT_TABLE table=test-table [id=9ddce4892e4149409e95fb7731355f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:56:15.332398 377 tablet_service.cc:1467] Processing CreateTablet for tablet 215b74780cda4c20a79900da4b80525c (DEFAULT_TABLE table=test-table [id=9ddce4892e4149409e95fb7731355f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:56:15.333511 377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215b74780cda4c20a79900da4b80525c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:15.333511 32711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215b74780cda4c20a79900da4b80525c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:15.333518 511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215b74780cda4c20a79900da4b80525c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:15.340091 594 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Bootstrap starting.
I20260925 00:56:15.341413 596 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Bootstrap starting.
I20260925 00:56:15.341413 595 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap starting.
I20260925 00:56:15.342344 594 tablet_bootstrap.cc:654] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:15.343425 594 log.cc:824] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:15.343588 595 tablet_bootstrap.cc:654] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:15.344028 596 tablet_bootstrap.cc:654] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:15.344460 595 log.cc:824] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:15.344837 596 log.cc:824] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:15.345664 594 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: No bootstrap required, opened a new log
I20260925 00:56:15.345865 594 ts_tablet_manager.cc:1397] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:56:15.346603 595 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: No bootstrap required, opened a new log
I20260925 00:56:15.346773 595 ts_tablet_manager.cc:1397] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:56:15.346894 596 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: No bootstrap required, opened a new log
I20260925 00:56:15.347072 596 ts_tablet_manager.cc:1397] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:56:15.349798 594 raft_consensus.cc:348] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.350095 594 raft_consensus.cc:374] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:15.350181 594 raft_consensus.cc:729] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae007bbaca2943a2b30a9e51ebbc5eec, State: Initialized, Role: FOLLOWER
I20260925 00:56:15.350072 595 raft_consensus.cc:348] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.350315 595 raft_consensus.cc:374] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:15.350368 595 raft_consensus.cc:729] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6363f966d1dc41c78977b86385f7e8c9, State: Initialized, Role: FOLLOWER
I20260925 00:56:15.350666 594 consensus_queue.cc:260] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.350672 596 raft_consensus.cc:348] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.350783 595 consensus_queue.cc:260] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.350975 596 raft_consensus.cc:374] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:15.351049 596 raft_consensus.cc:729] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2e6a9ad7754b5b8070269fbf06bd5a, State: Initialized, Role: FOLLOWER
I20260925 00:56:15.351476 309 heartbeater.cc:503] Master 127.16.202.62:36881 was elected leader, sending a full tablet report...
I20260925 00:56:15.351459 596 consensus_queue.cc:260] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.351747 594 ts_tablet_manager.cc:1428] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:56:15.351756 595 ts_tablet_manager.cc:1428] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 00:56:15.352114 577 heartbeater.cc:503] Master 127.16.202.62:36881 was elected leader, sending a full tablet report...
I20260925 00:56:15.352335 596 ts_tablet_manager.cc:1428] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:56:15.352725 443 heartbeater.cc:503] Master 127.16.202.62:36881 was elected leader, sending a full tablet report...
I20260925 00:56:15.419016 600 raft_consensus.cc:482] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:56:15.419236 600 raft_consensus.cc:504] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.420542 600 leader_election.cc:288] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827), 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183)
I20260925 00:56:15.424283 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.420946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:41969 (local address 127.16.202.2:32827)
0925 00:56:15.421057 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.421061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.421076 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:56:15.421198 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.421201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.421251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.421314 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:15.421723 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.422368 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.423367 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.423770 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.423887 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.423944 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.424109 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.424184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:56:15.424281 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.420757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32827 (local address 127.16.202.3:41969)
0925 00:56:15.420958 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.420973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.421065 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.421438 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.421442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.421460 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:15.421568 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.421574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.422469 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.422472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.423239 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.423248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.423884 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.423888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.423989 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.424008 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.424072 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.424145 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 00:56:15.424929 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.421187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39183 (local address 127.16.202.3:38501)
0925 00:56:15.421569 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.421582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:15.421672 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:15.422007 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:15.422010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:15.422030 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:15.422145 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.422151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.422952 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.422957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.423753 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:15.423762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.424546 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.424549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:15.424651 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.424698 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 00:56:15.424769 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 00:56:15.424822 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
I20260925 00:56:15.425015 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.421197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:38501 (local address 127.16.202.1:39183)
0925 00:56:15.421318 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.421322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.421674 (+ 352us) server_negotiation.cc:407] Connection header received
0925 00:56:15.421753 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.421756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.421809 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.421900 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:15.422264 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.422817 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.423897 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.424308 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.424447 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.424508 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.424800 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.424884 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:56:15.425057 397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215b74780cda4c20a79900da4b80525c" candidate_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" is_pre_election: true
I20260925 00:56:15.425413 397 raft_consensus.cc:2454] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae007bbaca2943a2b30a9e51ebbc5eec in term 0.
I20260925 00:56:15.425642 32731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215b74780cda4c20a79900da4b80525c" candidate_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6363f966d1dc41c78977b86385f7e8c9" is_pre_election: true
I20260925 00:56:15.425927 32731 raft_consensus.cc:2454] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae007bbaca2943a2b30a9e51ebbc5eec in term 0.
I20260925 00:56:15.425990 465 leader_election.cc:302] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae007bbaca2943a2b30a9e51ebbc5eec, dd2e6a9ad7754b5b8070269fbf06bd5a; no voters:
I20260925 00:56:15.426288 600 raft_consensus.cc:2781] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:56:15.426399 600 raft_consensus.cc:482] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:56:15.426438 600 raft_consensus.cc:3037] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:15.427464 600 raft_consensus.cc:504] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.427809 600 leader_election.cc:288] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [CANDIDATE]: Term 1 election: Requested vote from peers dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827), 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183)
I20260925 00:56:15.428159 397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215b74780cda4c20a79900da4b80525c" candidate_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
I20260925 00:56:15.428188 32731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215b74780cda4c20a79900da4b80525c" candidate_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6363f966d1dc41c78977b86385f7e8c9"
I20260925 00:56:15.428303 32731 raft_consensus.cc:3037] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:15.428308 397 raft_consensus.cc:3037] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:15.429409 32731 raft_consensus.cc:2454] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae007bbaca2943a2b30a9e51ebbc5eec in term 1.
I20260925 00:56:15.429448 397 raft_consensus.cc:2454] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae007bbaca2943a2b30a9e51ebbc5eec in term 1.
I20260925 00:56:15.429710 464 leader_election.cc:302] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9, ae007bbaca2943a2b30a9e51ebbc5eec; no voters:
I20260925 00:56:15.429951 600 raft_consensus.cc:2781] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:56:15.430154 600 raft_consensus.cc:686] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 LEADER]: Becoming Leader. State: Replica: ae007bbaca2943a2b30a9e51ebbc5eec, State: Running, Role: LEADER
I20260925 00:56:15.430513 600 consensus_queue.cc:237] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:15.433863 32585 catalog_manager.cc:5495] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec reported cstate change: term changed from 0 to 1, leader changed from <none> to ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3). New cstate: current_term: 1 leader_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } health_report { overall_health: HEALTHY } } }
I20260925 00:56:15.454910 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.451428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.16.202.3:38389)
0925 00:56:15.451582 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:15.451587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:15.451600 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:56:15.451675 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:15.451677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:15.451729 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:15.451822 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:56:15.452075 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.452584 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.453465 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:15.453674 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:15.453761 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:15.454085 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:56:15.454520 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:56:15.454630 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:15.454700 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 00:56:15.454761 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:56:15.460906 32731 raft_consensus.cc:1261] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Refusing update from remote peer ae007bbaca2943a2b30a9e51ebbc5eec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:56:15.460927 397 raft_consensus.cc:1261] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Refusing update from remote peer ae007bbaca2943a2b30a9e51ebbc5eec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:56:15.461616 600 consensus_queue.cc:1035] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:56:15.461817 604 consensus_queue.cc:1035] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:56:15.469934 608 mvcc.cc:205] Tried to move back new op lower bound from 7333059688280231936 to 7333059688166313984. Current Snapshot: MvccSnapshot[applied={T|T < 7333059688280231936}]
I20260925 00:56:15.478817 32586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:56:15.479020 32586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:56:15.481837 32586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Sending DeleteTablet for 3 replicas of tablet 215b74780cda4c20a79900da4b80525c
I20260925 00:56:15.482770 32711 tablet_service.cc:1514] Processing DeleteTablet for tablet 215b74780cda4c20a79900da4b80525c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:15 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:48546
I20260925 00:56:15.482904 377 tablet_service.cc:1514] Processing DeleteTablet for tablet 215b74780cda4c20a79900da4b80525c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:15 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:34676
I20260925 00:56:15.482935 511 tablet_service.cc:1514] Processing DeleteTablet for tablet 215b74780cda4c20a79900da4b80525c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:15 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:52230
I20260925 00:56:15.483574 618 tablet_replica.cc:330] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: stopping tablet replica
I20260925 00:56:15.483598 619 tablet_replica.cc:330] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: stopping tablet replica
I20260925 00:56:15.483805 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 32648
I20260925 00:56:15.483877 617 tablet_replica.cc:330] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: stopping tablet replica
I20260925 00:56:15.484369 619 raft_consensus.cc:2229] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:56:15.484875 619 raft_consensus.cc:2258] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:56:15.484913 618 raft_consensus.cc:2229] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:56:15.485267 618 raft_consensus.cc:2258] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:56:15.486606 619 ts_tablet_manager.cc:1905] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:56:15.486807 618 ts_tablet_manager.cc:1905] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:56:15.489566 618 ts_tablet_manager.cc:1918] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:56:15.489641 619 ts_tablet_manager.cc:1918] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:56:15.489773 619 log.cc:1196] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/215b74780cda4c20a79900da4b80525c
I20260925 00:56:15.489792 618 log.cc:1196] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/215b74780cda4c20a79900da4b80525c
I20260925 00:56:15.490134 619 ts_tablet_manager.cc:1939] T 215b74780cda4c20a79900da4b80525c P ae007bbaca2943a2b30a9e51ebbc5eec: Deleting consensus metadata
I20260925 00:56:15.490236 618 ts_tablet_manager.cc:1939] T 215b74780cda4c20a79900da4b80525c P dd2e6a9ad7754b5b8070269fbf06bd5a: Deleting consensus metadata
I20260925 00:56:15.491660 32571 catalog_manager.cc:4841] TS dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827): tablet 215b74780cda4c20a79900da4b80525c (table test-table [id=9ddce4892e4149409e95fb7731355f13]) successfully deleted
I20260925 00:56:15.491820 32573 catalog_manager.cc:4841] TS ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389): tablet 215b74780cda4c20a79900da4b80525c (table test-table [id=9ddce4892e4149409e95fb7731355f13]) successfully deleted
W20260925 00:56:15.492105 32570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:39183 (error 108)
I20260925 00:56:15.492238 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 314
I20260925 00:56:15.497843 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:15.497393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39183 (local address 127.0.0.1:48562)
0925 00:56:15.497667 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:15.497741 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:39183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
W20260925 00:56:15.498111 32570 catalog_manager.cc:4584] TS 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 215b74780cda4c20a79900da4b80525c: Network error: Client connection negotiation failed: client connection to 127.16.202.1:39183: connect: Connection refused (error 111)
I20260925 00:56:15.498677 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 448
I20260925 00:56:15.504195 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 32554
I20260925 00:56:35.510250 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36881
--webserver_interface=127.16.202.62
--webserver_port=37269
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:56:35.629158 621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:35.629467 621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:35.629513 621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:35.633882 621 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 00:56:35.633957 621 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 00:56:35.633973 621 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 00:56:35.633986 621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:56:35.634004 621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:35.634021 621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:35.634037 621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:56:35.634056 621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:56:35.639715 621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=37269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:35.640897 621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:35.641950 621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:35.655735 627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:35.655766 630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:35.655874 621 server_base.cc:1034] running on GCE node
W20260925 00:56:35.655767 628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:35.656724 621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:35.658030 621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:35.659204 621 hybrid_clock.cc:648] HybridClock initialized: now 1790297795659172 us; error 47 us; skew 500 ppm
Sep 25 00:56:35 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:35.665956 621 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:56:35.667093 621 webserver.cc:460] Webserver started at http://127.16.202.62:37269/ using document root <none> and password file <none>
I20260925 00:56:35.667758 621 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:35.667852 621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:35.672320 621 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:56:35.674983 637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:35.676630 621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 00:56:35.676805 621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "14953451e860409b980090cb833f7899"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "c7fa245aa6b6558d01bd0d8a29744f80"
server_key_iv: "920f09128ad04e393d7f947f00b71ffc"
server_key_version: "encryptionkey@0"
I20260925 00:56:35.677471 621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:35.702611 621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:35.705843 621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:35.706036 621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:35.713167 621 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36881
I20260925 00:56:35.713420 689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36881 every 8 connection(s)
I20260925 00:56:35.714354 621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:56:35.718384 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 621
I20260925 00:56:35.718859 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:39183
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:36881
--webserver_port=38807
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:56:35.719193 690 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:56:35.721308 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap starting.
I20260925 00:56:35.739418 690 log.cc:824] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:35.748577 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:56:35.748998 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap complete.
I20260925 00:56:35.752949 690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:35.753413 690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14953451e860409b980090cb833f7899, State: Initialized, Role: FOLLOWER
I20260925 00:56:35.753952 690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:35.754069 690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:56:35.754138 690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:56:35.754233 690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:56:35.756201 690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:35.756528 690 leader_election.cc:302] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14953451e860409b980090cb833f7899; no voters:
I20260925 00:56:35.756866 690 leader_election.cc:288] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:56:35.756994 694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:56:35.757218 694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 LEADER]: Becoming Leader. State: Replica: 14953451e860409b980090cb833f7899, State: Running, Role: LEADER
I20260925 00:56:35.757525 694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:35.758289 690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:56:35.758641 694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14953451e860409b980090cb833f7899. Latest consensus state: current_term: 2 leader_uuid: "14953451e860409b980090cb833f7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } } }
I20260925 00:56:35.758778 694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: This master's current role is: LEADER
I20260925 00:56:35.759279 695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "14953451e860409b980090cb833f7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } } }
I20260925 00:56:35.759474 695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: This master's current role is: LEADER
I20260925 00:56:35.759544 698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:56:35.764622 698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:56:35.765569 698 catalog_manager.cc:1259] Loaded cluster ID: 7923c4d31e8b42c68c87003181cca5a6
I20260925 00:56:35.765642 698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:56:35.768165 698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:56:35.768971 698 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Loaded TSK: 0
I20260925 00:56:35.781859 698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Generated new TSK 1
I20260925 00:56:35.782205 698 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:56:35.837332 692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:35.837616 692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:35.837682 692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:35.841816 692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:35.841898 692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:35.841984 692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:56:35.847499 692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:39183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=38807
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:35.848557 692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:35.849592 692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:35.861661 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:35.861631 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:35.861631 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:35.861862 692 server_base.cc:1034] running on GCE node
I20260925 00:56:35.862407 692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:35.863121 692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:35.864298 692 hybrid_clock.cc:648] HybridClock initialized: now 1790297795864259 us; error 60 us; skew 500 ppm
Sep 25 00:56:35 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:35.867357 692 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:56:35.868408 692 webserver.cc:460] Webserver started at http://127.16.202.1:38807/ using document root <none> and password file <none>
I20260925 00:56:35.869015 692 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:35.869102 692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:35.873273 692 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:56:35.875663 727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:35.877233 692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:35.877369 692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "6363f966d1dc41c78977b86385f7e8c9"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "b70bb0ad3a76ebfc4cdf3b535f7700f1"
server_key_iv: "2c31ed484ab9952fba5a4ea43fc767fb"
server_key_version: "encryptionkey@0"
I20260925 00:56:35.877959 692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:35.893868 692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:35.897441 692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:35.897655 692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:35.898344 692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:35.899564 734 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:56:35.903759 692 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:56:35.903839 692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 00:56:35.903890 692 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:56:35.905808 692 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:56:35.905879 692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260925 00:56:35.906172 734 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap starting.
I20260925 00:56:35.916389 692 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:39183
I20260925 00:56:35.916422 841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:39183 every 8 connection(s)
I20260925 00:56:35.917773 692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:56:35.926245 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 692
I20260925 00:56:35.926721 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:32827
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:36881
--webserver_port=38089
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:56:35.926721 734 log.cc:824] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:56:35 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:35.934913 734 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:56:35.935351 734 tablet_bootstrap.cc:492] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap complete.
I20260925 00:56:35.935688 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:35.919999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:35301 (local address 127.16.202.62:36881)
0925 00:56:35.920353 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:35.920374 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:35.920750 (+ 376us) server_negotiation.cc:407] Connection header received
0925 00:56:35.921956 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:35.921968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:35.922305 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:35.922628 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:35.923837 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.924883 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.926103 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.926630 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.926796 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:35.929819 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:35.929826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:35.929844 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:35.929880 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:35.931917 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:35.932560 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:35.932566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:35.932572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:35.932746 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:35.933124 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:35.933127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:35.933129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:35.933580 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:35.933703 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:35.934365 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 00:56:35.934678 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
I20260925 00:56:35.935724 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:35.920054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:35301)
0925 00:56:35.920579 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:35.920623 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:35.921472 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:35.922737 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:35.922746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:35.923164 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:35.923658 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:35.923674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.925043 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.925046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:35.925979 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:35.925990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.926768 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.926772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:35.927140 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:35.927804 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:35.927825 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:35.929600 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:35.932050 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:35.932057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:35.932070 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:35.932421 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:35.932841 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:35.932845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:35.932846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:35.933024 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:35.933697 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:35.933703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:35.934015 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 00:56:35.934230 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 00:56:35.934598 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":158,"threads_started":1}
I20260925 00:56:35.936580 734 ts_tablet_manager.cc:1397] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.009s
I20260925 00:56:35.938683 842 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36881
I20260925 00:56:35.938932 842 heartbeater.cc:464] Registering TS with master...
I20260925 00:56:35.939507 842 heartbeater.cc:511] Master 127.16.202.62:36881 requested a full tablet report, sending...
I20260925 00:56:35.939291 734 raft_consensus.cc:348] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:35.939739 734 raft_consensus.cc:729] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6363f966d1dc41c78977b86385f7e8c9, State: Initialized, Role: FOLLOWER
I20260925 00:56:35.940233 734 consensus_queue.cc:260] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } } peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } }
I20260925 00:56:35.941208 734 ts_tablet_manager.cc:1428] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260925 00:56:35.943019 654 ts_manager.cc:194] Registered new tserver with Master: 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183)
I20260925 00:56:35.946858 654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:35301
I20260925 00:56:35.949855 842 heartbeater.cc:503] Master 127.16.202.62:36881 was elected leader, sending a full tablet report...
Sep 25 00:56:35 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.1@KRBTEST.COM
I20260925 00:56:35.960088 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:35.946371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39183 (local address 127.0.0.1:46354)
0925 00:56:35.947352 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:35.947422 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:35.947596 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:35.948443 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:35.948449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:35.949004 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:35.949148 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:35.949158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.950568 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.950571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:35.951580 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:35.951592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.952322 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.952326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:35.952460 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:35.953112 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:35.953123 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:35.954818 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:35.957529 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:35.957568 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:35.957576 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:35.958095 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:35.958518 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:35.958546 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:35.958548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:35.958686 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:35.959369 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:35.959376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:35.959528 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 00:56:35.959651 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 00:56:35.959900 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":129,"threads_started":1}
I20260925 00:56:35.960871 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:35.946468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.16.202.1:39183)
0925 00:56:35.947921 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:35.947932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:35.947955 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:56:35.948045 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:35.948051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:35.948268 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:35.948410 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:35.949283 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.950450 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.951854 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:35.952199 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:35.952295 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:35.955016 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:35.955031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:35.955044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:35.955073 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:35.957315 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:35.958286 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:35.958293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:35.958295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:35.958405 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:35.958812 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:35.958816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:35.958818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:35.959231 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:35.960367 (+ 1136us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:35.960491 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 00:56:35.960667 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":135,"threads_started":1}
I20260925 00:56:35.961970 776 tablet_service.cc:1514] Processing DeleteTablet for tablet 215b74780cda4c20a79900da4b80525c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:15 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260925 00:56:35.962965 851 tablet_replica.cc:330] stopping tablet replica
I20260925 00:56:35.963323 851 raft_consensus.cc:2229] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:56:35.963617 851 raft_consensus.cc:2258] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:56:35.964861 851 ts_tablet_manager.cc:1905] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:56:35.969602 851 ts_tablet_manager.cc:1918] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:56:35.969735 851 log.cc:1196] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal/wals/215b74780cda4c20a79900da4b80525c
I20260925 00:56:35.970161 851 ts_tablet_manager.cc:1939] T 215b74780cda4c20a79900da4b80525c P 6363f966d1dc41c78977b86385f7e8c9: Deleting consensus metadata
I20260925 00:56:35.972754 638 catalog_manager.cc:4841] TS 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183): tablet 215b74780cda4c20a79900da4b80525c (table test-table [id=9ddce4892e4149409e95fb7731355f13]) successfully deleted
W20260925 00:56:36.060575 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:36.060906 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:36.060952 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:36.065073 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:36.065146 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:36.065222 847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:56:36.070740 847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:32827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=38089
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:36.071887 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:36.072968 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:36.082636 856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:36.082720 859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:36.082720 857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:36.082871 847 server_base.cc:1034] running on GCE node
I20260925 00:56:36.083325 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:36.083985 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:36.085163 847 hybrid_clock.cc:648] HybridClock initialized: now 1790297796085143 us; error 31 us; skew 500 ppm
Sep 25 00:56:36 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297796, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:36.087805 847 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:56:36.088760 847 webserver.cc:460] Webserver started at http://127.16.202.2:38089/ using document root <none> and password file <none>
I20260925 00:56:36.089318 847 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:36.089404 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:36.093425 847 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:56:36.095630 866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:36.096985 847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:56:36.097096 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "4b2a3b016faaf7511a8c55fac8897691"
server_key_iv: "e396739908f4da63d9b4cbaa073af770"
server_key_version: "encryptionkey@0"
I20260925 00:56:36.097637 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:36.123628 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:36.126902 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:36.127085 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:36.127719 847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:36.128561 847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:36.128607 847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:36.128659 847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:36.128712 847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:36.137282 847 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:32827
I20260925 00:56:36.137365 979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:32827 every 8 connection(s)
I20260925 00:56:36.138306 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:56:36.146729 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 847
I20260925 00:56:36.147176 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:38389
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:36881
--webserver_port=39563
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:56:36 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297796, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:36.152427 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.140153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:59745 (local address 127.16.202.62:36881)
0925 00:56:36.140296 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.140301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.141212 (+ 911us) server_negotiation.cc:407] Connection header received
0925 00:56:36.142541 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.142544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.142598 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.142676 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:36.143859 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.144390 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.145671 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.145919 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.146022 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.148809 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:36.148813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:36.148816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:36.148847 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:36.150273 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.150855 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.150857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.150858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.150924 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.151284 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.151287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.151288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.151466 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:36.151565 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.152190 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.152304 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:56:36.153635 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.140569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:59745)
0925 00:56:36.141056 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.141099 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:36.142291 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:36.142843 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:36.142850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:36.143224 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:36.143712 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.143725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.144548 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.144551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.145531 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.145542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.146099 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.146102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.146257 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.146961 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:36.146978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:36.148605 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:36.150390 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:36.150396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:36.150413 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:36.150763 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:36.150999 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:36.151002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:36.151003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:36.151186 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:36.151564 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:36.151571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:36.151911 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 00:56:36.152126 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 00:56:36.152499 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":109,"threads_started":1}
I20260925 00:56:36.155007 980 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36881
I20260925 00:56:36.155346 980 heartbeater.cc:464] Registering TS with master...
I20260925 00:56:36.156009 980 heartbeater.cc:511] Master 127.16.202.62:36881 requested a full tablet report, sending...
I20260925 00:56:36.157204 654 ts_manager.cc:194] Registered new tserver with Master: dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
I20260925 00:56:36.157776 654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:59745
W20260925 00:56:36.260643 983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:36.260946 983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:36.261010 983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:36.265095 983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:36.265161 983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:36.265241 983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:56:36.270649 983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:38389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=39563
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:36.271787 983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:36.272846 983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:36.282812 989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:36.282806 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:36.282812 988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:36.283160 983 server_base.cc:1034] running on GCE node
I20260925 00:56:36.283610 983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:36.284245 983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:36.285418 983 hybrid_clock.cc:648] HybridClock initialized: now 1790297796285396 us; error 33 us; skew 500 ppm
Sep 25 00:56:36 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297796, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:36.288151 983 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:56:36.289208 983 webserver.cc:460] Webserver started at http://127.16.202.3:39563/ using document root <none> and password file <none>
I20260925 00:56:36.289776 983 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:36.289860 983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:36.294296 983 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:56:36.296653 998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:36.298106 983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:36.298221 983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "ae007bbaca2943a2b30a9e51ebbc5eec"
format_stamp: "Formatted at 2026-09-25 00:56:15 on dist-test-slave-rpk1"
server_key: "9673cc3bb90feffc1bde3c35b3805bbe"
server_key_iv: "6535b470be4030f4b1458dad3f4a6df9"
server_key_version: "encryptionkey@0"
I20260925 00:56:36.298842 983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:36.323638 983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:36.327980 983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:36.328163 983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:36.328831 983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:36.329799 983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:36.329859 983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:36.329906 983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:36.329937 983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:36.338567 983 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:38389
I20260925 00:56:36.338591 1111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:38389 every 8 connection(s)
I20260925 00:56:36.339617 983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:56:36.344578 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 983
Sep 25 00:56:36 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297796, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:56:36.355391 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.345273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.16.202.62:36881)
0925 00:56:36.346141 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.346144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.346152 (+ 8us) server_negotiation.cc:407] Connection header received
0925 00:56:36.346176 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.346178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.346212 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.346272 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:36.347199 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.347889 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.349025 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.349240 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.349316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.350931 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:36.350934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:36.350936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:36.350967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:36.353313 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.353880 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.353883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.353884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.354001 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.354361 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.354364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.354365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.354706 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:36.355041 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.355119 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.355232 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"spinlock_wait_cycles":1792,"thread_start_us":149,"threads_started":1}
I20260925 00:56:36.355852 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.341452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:47775 (local address 127.16.202.62:36881)
0925 00:56:36.341594 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.341598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.342548 (+ 950us) server_negotiation.cc:407] Connection header received
0925 00:56:36.343825 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.343828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.343886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.343960 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:36.345401 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.345951 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.347429 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.347797 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.347904 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.350890 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:36.350895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:36.350897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:36.350927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:36.353313 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.353976 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.353979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.353980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.354048 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.354558 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.354562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.354564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.354740 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:36.354816 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.355590 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.355711 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:56:36.356913 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.341890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:47775)
0925 00:56:36.342349 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.342393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:36.343616 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:36.344170 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:36.344177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:36.344561 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:36.345240 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.345262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.346074 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.346077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.347313 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.347322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.347933 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.347936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.348165 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.348832 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:36.348853 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:36.350664 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:36.353460 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:36.353467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:36.353488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:36.353874 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:36.354152 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:36.354155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:36.354157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:36.354387 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:36.354872 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:36.354880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:36.355243 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 00:56:36.355468 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 00:56:36.355846 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260925 00:56:36.358040 1112 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36881
I20260925 00:56:36.358304 1112 heartbeater.cc:464] Registering TS with master...
I20260925 00:56:36.358784 1112 heartbeater.cc:511] Master 127.16.202.62:36881 requested a full tablet report, sending...
I20260925 00:56:36.359754 654 ts_manager.cc:194] Registered new tserver with Master: ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389)
I20260925 00:56:36.360324 654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:47775
I20260925 00:56:36.369527 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:56:36.373651 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.370114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.16.202.62:36881)
0925 00:56:36.370254 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.370259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.370387 (+ 128us) server_negotiation.cc:407] Connection header received
0925 00:56:36.370559 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.370561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.370677 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.370760 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:56:36.370961 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.371571 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.372484 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.372749 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.372832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.373131 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:56:36.373286 (+ 155us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 00:56:36.373536 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47166: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 00:56:36.373744 844 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47166: token expired
W20260925 00:56:36.374190 581 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.202.62:36881)
I20260925 00:56:36.381162 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.374551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.16.202.62:36881)
0925 00:56:36.374673 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.374677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.374689 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:56:36.374791 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.374794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.374838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.374902 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:36.375505 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.375993 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.376830 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.377030 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.377121 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.378322 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:36.378325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:36.378327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:36.378352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:36.379526 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.379922 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.379925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.379926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.380009 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:36.380275 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:36.380278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:36.380280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:36.380469 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:36.380591 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.380876 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.380996 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:56:36.383884 654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:56:36.387236 654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:56:36.395011 776 tablet_service.cc:1467] Processing CreateTablet for tablet 269894963a594b2ea73eca2fb74b43ed (DEFAULT_TABLE table=test-table [id=89c1a8f44367469589ea6e1cfa8876cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:56:36.395720 776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269894963a594b2ea73eca2fb74b43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:36.398640 734 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap starting.
I20260925 00:56:36.399094 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.393829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:38389 (local address 127.0.0.1:52106)
0925 00:56:36.393992 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.394006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:36.394083 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:36.395081 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:36.395084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:36.395104 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:36.395215 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.395222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.396341 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.396344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.397524 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.397535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.398658 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.398662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.398763 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.398820 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 00:56:36.398870 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 00:56:36.398986 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":1}
I20260925 00:56:36.399567 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.394235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32827 (local address 127.0.0.1:52802)
0925 00:56:36.394536 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.394549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:36.394638 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:36.395415 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:36.395418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:36.395436 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:36.395531 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.395537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.397029 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.397032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.397979 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.397988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.398868 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.398871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.399272 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.399298 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:56:36.399368 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:56:36.399439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":268,"thread_start_us":87,"threads_started":1}
I20260925 00:56:36.399608 734 tablet_bootstrap.cc:654] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:36.399686 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.394410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.16.202.2:32827)
0925 00:56:36.394887 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.394898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.394920 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:56:36.395008 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.395014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.395231 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.395673 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:36.395844 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.396907 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.398092 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.398734 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.399008 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.399201 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.399415 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.399533 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":163,"threads_started":1}
I20260925 00:56:36.399796 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.394014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.16.202.3:38389)
0925 00:56:36.394599 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.394611 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.394636 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:56:36.394719 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.394725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.394925 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.395087 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:36.395343 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.396228 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.397658 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.398523 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.399350 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.399513 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.399565 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.399662 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":143,"threads_started":1}
I20260925 00:56:36.401598 734 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: No bootstrap required, opened a new log
I20260925 00:56:36.401696 734 ts_tablet_manager.cc:1397] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:56:36.402175 734 raft_consensus.cc:348] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.402299 734 raft_consensus.cc:374] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:36.402339 734 raft_consensus.cc:729] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6363f966d1dc41c78977b86385f7e8c9, State: Initialized, Role: FOLLOWER
I20260925 00:56:36.402490 734 consensus_queue.cc:260] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.402853 734 ts_tablet_manager.cc:1428] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:56:36.402695 1046 tablet_service.cc:1467] Processing CreateTablet for tablet 269894963a594b2ea73eca2fb74b43ed (DEFAULT_TABLE table=test-table [id=89c1a8f44367469589ea6e1cfa8876cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:56:36.402691 914 tablet_service.cc:1467] Processing CreateTablet for tablet 269894963a594b2ea73eca2fb74b43ed (DEFAULT_TABLE table=test-table [id=89c1a8f44367469589ea6e1cfa8876cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:56:36.403765 1046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269894963a594b2ea73eca2fb74b43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:36.403764 914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269894963a594b2ea73eca2fb74b43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:56:36.410543 1125 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Bootstrap starting.
I20260925 00:56:36.411041 1126 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Bootstrap starting.
I20260925 00:56:36.412712 1126 tablet_bootstrap.cc:654] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:36.412968 1125 tablet_bootstrap.cc:654] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Neither blocks nor log segments found. Creating new log.
I20260925 00:56:36.413800 1125 log.cc:824] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:36.413851 1126 log.cc:824] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:36.415570 1125 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: No bootstrap required, opened a new log
I20260925 00:56:36.415577 1126 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: No bootstrap required, opened a new log
I20260925 00:56:36.415745 1126 ts_tablet_manager.cc:1397] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:56:36.415745 1125 ts_tablet_manager.cc:1397] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:56:36.418596 1126 raft_consensus.cc:348] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.418597 1125 raft_consensus.cc:348] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.418834 1126 raft_consensus.cc:374] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:36.418891 1126 raft_consensus.cc:729] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae007bbaca2943a2b30a9e51ebbc5eec, State: Initialized, Role: FOLLOWER
W20260925 00:56:36.419042 843 tablet.cc:2369] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:56:36.419322 1126 consensus_queue.cc:260] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.418841 1125 raft_consensus.cc:374] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:56:36.419495 1125 raft_consensus.cc:729] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2e6a9ad7754b5b8070269fbf06bd5a, State: Initialized, Role: FOLLOWER
I20260925 00:56:36.419890 1125 consensus_queue.cc:260] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.420022 1112 heartbeater.cc:503] Master 127.16.202.62:36881 was elected leader, sending a full tablet report...
I20260925 00:56:36.420315 1126 ts_tablet_manager.cc:1428] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:56:36.420584 980 heartbeater.cc:503] Master 127.16.202.62:36881 was elected leader, sending a full tablet report...
I20260925 00:56:36.420900 1125 ts_tablet_manager.cc:1428] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:56:36.579408 848 raft_consensus.cc:482] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:56:36.579778 848 raft_consensus.cc:504] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.581094 848 leader_election.cc:288] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389), dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
I20260925 00:56:36.586711 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.581618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:59487 (local address 127.16.202.3:38389)
0925 00:56:36.581720 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.581723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.581952 (+ 229us) server_negotiation.cc:407] Connection header received
0925 00:56:36.582050 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.582053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.582103 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.582198 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:36.582574 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.583359 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.585598 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.586032 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.586171 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.586218 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.586494 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.586580 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 00:56:36.586723 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.581418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:38389 (local address 127.16.202.1:59487)
0925 00:56:36.581876 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.581888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:36.581961 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:36.582220 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:36.582223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:36.582267 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:36.582438 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.582446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.583506 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.583511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.585353 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.585369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.586191 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.586196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.586300 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.586368 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 00:56:36.586457 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 00:56:36.586550 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":201,"spinlock_wait_cycles":8192,"thread_start_us":68,"threads_started":1}
I20260925 00:56:36.587361 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.581485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34877 (local address 127.16.202.2:32827)
0925 00:56:36.581627 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.581631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.581896 (+ 265us) server_negotiation.cc:407] Connection header received
0925 00:56:36.582050 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.582053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.582105 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.582198 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:56:36.582603 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.583359 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.586261 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.586677 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.586787 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.586846 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.587179 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.587261 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:56:36.587360 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.581390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32827 (local address 127.16.202.1:34877)
0925 00:56:36.581772 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.581789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:36.581924 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:36.582220 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:36.582223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:36.582266 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:56:36.582466 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.582472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.583506 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.583511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.585353 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:36.585369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.586930 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.586934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:36.587045 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.587067 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:56:36.587143 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:56:36.587221 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":301,"spinlock_wait_cycles":28288,"thread_start_us":78,"threads_started":1}
I20260925 00:56:36.587363 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269894963a594b2ea73eca2fb74b43ed" candidate_uuid: "6363f966d1dc41c78977b86385f7e8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" is_pre_election: true
I20260925 00:56:36.587723 1066 raft_consensus.cc:2454] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6363f966d1dc41c78977b86385f7e8c9 in term 0.
I20260925 00:56:36.587930 934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269894963a594b2ea73eca2fb74b43ed" candidate_uuid: "6363f966d1dc41c78977b86385f7e8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" is_pre_election: true
I20260925 00:56:36.588228 934 raft_consensus.cc:2454] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6363f966d1dc41c78977b86385f7e8c9 in term 0.
I20260925 00:56:36.588263 731 leader_election.cc:302] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9, ae007bbaca2943a2b30a9e51ebbc5eec; no voters:
I20260925 00:56:36.588574 848 raft_consensus.cc:2781] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:56:36.588663 848 raft_consensus.cc:482] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:56:36.588750 848 raft_consensus.cc:3037] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:36.589835 848 raft_consensus.cc:504] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.590240 848 leader_election.cc:288] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 1 election: Requested vote from peers ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389), dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
W20260925 00:56:36.590497 1113 tablet.cc:2369] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:56:36.590574 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269894963a594b2ea73eca2fb74b43ed" candidate_uuid: "6363f966d1dc41c78977b86385f7e8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec"
I20260925 00:56:36.590598 934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269894963a594b2ea73eca2fb74b43ed" candidate_uuid: "6363f966d1dc41c78977b86385f7e8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
I20260925 00:56:36.590696 1066 raft_consensus.cc:3037] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:36.590723 934 raft_consensus.cc:3037] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:56:36.591802 934 raft_consensus.cc:2454] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6363f966d1dc41c78977b86385f7e8c9 in term 1.
I20260925 00:56:36.591816 1066 raft_consensus.cc:2454] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6363f966d1dc41c78977b86385f7e8c9 in term 1.
I20260925 00:56:36.592139 731 leader_election.cc:302] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9, ae007bbaca2943a2b30a9e51ebbc5eec; no voters:
I20260925 00:56:36.592322 848 raft_consensus.cc:2781] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:56:36.592514 848 raft_consensus.cc:686] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6363f966d1dc41c78977b86385f7e8c9, State: Running, Role: LEADER
I20260925 00:56:36.592813 848 consensus_queue.cc:237] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:36.595659 654 catalog_manager.cc:5495] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "6363f966d1dc41c78977b86385f7e8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } health_report { overall_health: HEALTHY } } }
W20260925 00:56:36.639412 981 tablet.cc:2369] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:56:36.639953 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.635966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.16.202.1:39183)
0925 00:56:36.636263 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:36.636270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:36.636285 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:56:36.636380 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:36.636383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:36.636457 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:36.636602 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:56:36.636994 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.637672 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.638488 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:36.638729 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:36.638810 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:36.639104 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:56:36.639555 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:56:36.639674 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:36.639745 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 00:56:36.639800 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260925 00:56:36.644253 934 raft_consensus.cc:1261] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Refusing update from remote peer 6363f966d1dc41c78977b86385f7e8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:56:36.644254 1066 raft_consensus.cc:1261] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Refusing update from remote peer 6363f966d1dc41c78977b86385f7e8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:56:36.645033 1133 consensus_queue.cc:1035] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:56:36.645277 848 consensus_queue.cc:1035] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:56:36.654057 1139 mvcc.cc:205] Tried to move back new op lower bound from 7333059775047921664 to 7333059774846689280. Current Snapshot: MvccSnapshot[applied={T|T < 7333059775047921664}]
I20260925 00:56:36.663820 652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:56:36.664022 652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:56:36.666610 652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Sending DeleteTablet for 3 replicas of tablet 269894963a594b2ea73eca2fb74b43ed
I20260925 00:56:36.667528 776 tablet_service.cc:1514] Processing DeleteTablet for tablet 269894963a594b2ea73eca2fb74b43ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:36 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260925 00:56:36.667600 914 tablet_service.cc:1514] Processing DeleteTablet for tablet 269894963a594b2ea73eca2fb74b43ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:36 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:52802
I20260925 00:56:36.667595 1046 tablet_service.cc:1514] Processing DeleteTablet for tablet 269894963a594b2ea73eca2fb74b43ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:56:36 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:52106
I20260925 00:56:36.668288 1149 tablet_replica.cc:330] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: stopping tablet replica
I20260925 00:56:36.668290 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 692
I20260925 00:56:36.668711 1149 raft_consensus.cc:2229] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:56:36.669097 1149 raft_consensus.cc:2258] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:56:36.669587 1148 tablet_replica.cc:330] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: stopping tablet replica
I20260925 00:56:36.669901 1148 raft_consensus.cc:2229] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:56:36.670218 1148 raft_consensus.cc:2258] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:56:36.671492 1148 ts_tablet_manager.cc:1905] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:56:36.671509 1149 ts_tablet_manager.cc:1905] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:56:36.674213 1149 ts_tablet_manager.cc:1918] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:56:36.674227 1148 ts_tablet_manager.cc:1918] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:56:36.674321 1149 log.cc:1196] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/269894963a594b2ea73eca2fb74b43ed
I20260925 00:56:36.674321 1148 log.cc:1196] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/269894963a594b2ea73eca2fb74b43ed
I20260925 00:56:36.674623 1148 ts_tablet_manager.cc:1939] T 269894963a594b2ea73eca2fb74b43ed P dd2e6a9ad7754b5b8070269fbf06bd5a: Deleting consensus metadata
I20260925 00:56:36.674649 1149 ts_tablet_manager.cc:1939] T 269894963a594b2ea73eca2fb74b43ed P ae007bbaca2943a2b30a9e51ebbc5eec: Deleting consensus metadata
I20260925 00:56:36.675868 641 catalog_manager.cc:4841] TS ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389): tablet 269894963a594b2ea73eca2fb74b43ed (table test-table [id=89c1a8f44367469589ea6e1cfa8876cd]) successfully deleted
I20260925 00:56:36.676046 639 catalog_manager.cc:4841] TS dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827): tablet 269894963a594b2ea73eca2fb74b43ed (table test-table [id=89c1a8f44367469589ea6e1cfa8876cd]) successfully deleted
I20260925 00:56:36.676357 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 847
W20260925 00:56:36.676394 638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:39183 (error 108)
I20260925 00:56:36.677815 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:36.677479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:39183 (local address 127.0.0.1:46384)
0925 00:56:36.677663 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:36.677728 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:39183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260925 00:56:36.677974 638 catalog_manager.cc:4584] TS 6363f966d1dc41c78977b86385f7e8c9 (127.16.202.1:39183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 269894963a594b2ea73eca2fb74b43ed: Network error: Client connection negotiation failed: client connection to 127.16.202.1:39183: connect: Connection refused (error 111)
I20260925 00:56:36.681892 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 983
I20260925 00:56:36.687016 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 621
I20260925 00:56:56.693230 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36881
--webserver_interface=127.16.202.62
--webserver_port=37269
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:56:56.804096 1151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:56.804389 1151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:56.804430 1151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:56.808523 1151 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 00:56:56.808604 1151 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 00:56:56.808620 1151 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 00:56:56.808633 1151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:56:56.808651 1151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:56.808689 1151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:56.808705 1151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:56:56.808725 1151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:56:56.814316 1151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=37269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:56.815474 1151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:56.816727 1151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:56.825568 1156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:56.825562 1159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:56.825591 1157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:56.825613 1151 server_base.cc:1034] running on GCE node
I20260925 00:56:56.826328 1151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:56.827462 1151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:56.828644 1151 hybrid_clock.cc:648] HybridClock initialized: now 1790297816828625 us; error 30 us; skew 500 ppm
Sep 25 00:56:56 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:56.831439 1151 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:56:56.832337 1151 webserver.cc:460] Webserver started at http://127.16.202.62:37269/ using document root <none> and password file <none>
I20260925 00:56:56.832922 1151 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:56.832988 1151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:56.836969 1151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:56:56.839145 1166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:56.840502 1151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:56:56.840615 1151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "14953451e860409b980090cb833f7899"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "c7fa245aa6b6558d01bd0d8a29744f80"
server_key_iv: "920f09128ad04e393d7f947f00b71ffc"
server_key_version: "encryptionkey@0"
I20260925 00:56:56.841218 1151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:56.852241 1151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:56.855422 1151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:56.855610 1151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:56.862640 1151 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36881
I20260925 00:56:56.862919 1218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36881 every 8 connection(s)
I20260925 00:56:56.863708 1151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:56:56.867990 1219 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:56:56.869728 1219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap starting.
I20260925 00:56:56.870019 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1151
I20260925 00:56:56.870412 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:39183
--local_ip_for_outbound_sockets=127.16.202.1
--tserver_master_addrs=127.16.202.62:36881
--webserver_port=38807
--webserver_interface=127.16.202.1
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:56:56.887272 1219 log.cc:824] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:56.898949 1219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:56:56.899324 1219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Bootstrap complete.
I20260925 00:56:56.903367 1219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:56.903781 1219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14953451e860409b980090cb833f7899, State: Initialized, Role: FOLLOWER
I20260925 00:56:56.904316 1219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:56.904449 1219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:56:56.904520 1219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:56:56.904618 1219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 2 FOLLOWER]: Advancing to term 3
I20260925 00:56:56.906606 1219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:56.906950 1219 leader_election.cc:302] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14953451e860409b980090cb833f7899; no voters:
I20260925 00:56:56.907246 1219 leader_election.cc:288] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 00:56:56.907390 1223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 00:56:56.907646 1223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [term 3 LEADER]: Becoming Leader. State: Replica: 14953451e860409b980090cb833f7899, State: Running, Role: LEADER
I20260925 00:56:56.907967 1223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } }
I20260925 00:56:56.908398 1219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:56:56.909430 1224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "14953451e860409b980090cb833f7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } } }
I20260925 00:56:56.909565 1224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: This master's current role is: LEADER
I20260925 00:56:56.909865 1225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14953451e860409b980090cb833f7899. Latest consensus state: current_term: 3 leader_uuid: "14953451e860409b980090cb833f7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14953451e860409b980090cb833f7899" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36881 } } }
I20260925 00:56:56.909940 1225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899 [sys.catalog]: This master's current role is: LEADER
I20260925 00:56:56.910445 1232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:56:56.915637 1232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:56:56.916285 1232 catalog_manager.cc:1259] Loaded cluster ID: 7923c4d31e8b42c68c87003181cca5a6
I20260925 00:56:56.916348 1232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:56:56.918859 1232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:56:56.919600 1232 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Loaded TSK: 0
I20260925 00:56:56.919684 1232 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14953451e860409b980090cb833f7899: Loaded TSK: 1
F20260925 00:56:56.920449 1232 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9e6e6e9ad google::LogMessage::Fail() at ??:0
@ 0x7fb9e6e7188a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9e6e6e4de google::LogMessage::Flush() at ??:0
@ 0x7fb9e6e6fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9e9b3218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb9e9b2af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb9f17d37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb9f17ad99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb9f17ae178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fb9f17efdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb9f180e82b std::function<>::operator()() at ??:0
@ 0x7fb9f17adb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb9f17aef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9f17ab579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb9f17ef640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb9e90c03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9e90c0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb9e90c23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb9e90b1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9ea6f56db start_thread at ??:0
@ 0x7fb9e596c61f clone at ??:0
W20260925 00:56:57.016251 1221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:57.016657 1221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:57.016805 1221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:57.022068 1221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:57.022143 1221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:57.022231 1221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:56:57.028105 1221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:39183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=38807
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:57.029345 1221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:57.030419 1221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:57.040261 1267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:57.040275 1266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:57.040503 1221 server_base.cc:1034] running on GCE node
W20260925 00:56:57.040663 1269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:57.041126 1221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:57.041801 1221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:57.042985 1221 hybrid_clock.cc:648] HybridClock initialized: now 1790297817042973 us; error 33 us; skew 500 ppm
Sep 25 00:56:57 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:57.045765 1221 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:56:57.046689 1221 webserver.cc:460] Webserver started at http://127.16.202.1:38807/ using document root <none> and password file <none>
I20260925 00:56:57.047246 1221 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:57.047305 1221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:57.051442 1221 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 00:56:57.053777 1276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.055115 1221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:57.055222 1221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "6363f966d1dc41c78977b86385f7e8c9"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "b70bb0ad3a76ebfc4cdf3b535f7700f1"
server_key_iv: "2c31ed484ab9952fba5a4ea43fc767fb"
server_key_version: "encryptionkey@0"
I20260925 00:56:57.055799 1221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:57.066223 1221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:57.069530 1221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:57.069706 1221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:57.070376 1221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:57.071514 1283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:56:57.075593 1221 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:56:57.075654 1221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 00:56:57.075703 1221 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:56:57.077433 1221 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:56:57.077497 1221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 00:56:57.077782 1283 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap starting.
I20260925 00:56:57.087829 1221 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:39183
I20260925 00:56:57.088174 1390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:39183 every 8 connection(s)
I20260925 00:56:57.089169 1221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:56:57.097368 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1221
I20260925 00:56:57.097936 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:32827
--local_ip_for_outbound_sockets=127.16.202.2
--tserver_master_addrs=127.16.202.62:36881
--webserver_port=38089
--webserver_interface=127.16.202.2
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:56:57.097931 1283 log.cc:824] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:56:57.105296 1283 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:56:57.105679 1283 tablet_bootstrap.cc:492] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Bootstrap complete.
I20260925 00:56:57.106956 1283 ts_tablet_manager.cc:1397] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260925 00:56:57.110364 1283 raft_consensus.cc:348] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:57.110925 1283 raft_consensus.cc:729] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6363f966d1dc41c78977b86385f7e8c9, State: Initialized, Role: FOLLOWER
I20260925 00:56:57.111497 1283 consensus_queue.cc:260] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:57.112625 1283 ts_tablet_manager.cc:1428] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260925 00:56:57.138595 1393 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:56:57.091644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:52075)
0925 00:56:57.092451 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.092516 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:57.093665 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:57.137856 (+ 44191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":162,"threads_started":1}
W20260925 00:56:57.139196 1391 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:36881 (0 consecutive failures): Network error: Failed to ping master at 127.16.202.62:36881: Client connection negotiation failed: client connection to 127.16.202.62:36881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:56:57.139941 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.139681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:51973)
0925 00:56:57.139811 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.139878 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 00:56:57.140623 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.140405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:58705)
0925 00:56:57.140529 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.140571 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 00:56:57.140833 1391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:56:57.221573 1395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:57.221891 1395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:57.221956 1395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:57.226408 1395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:57.226487 1395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:57.226557 1395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:56:57.232600 1395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:32827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=38089
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:57.233819 1395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:57.234905 1395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:57.245565 1402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:57.245535 1404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:57.245534 1401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:57.245882 1395 server_base.cc:1034] running on GCE node
I20260925 00:56:57.246434 1395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:57.247236 1395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:57.248442 1395 hybrid_clock.cc:648] HybridClock initialized: now 1790297817248405 us; error 62 us; skew 500 ppm
Sep 25 00:56:57 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:57.251482 1395 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:56:57.252554 1395 webserver.cc:460] Webserver started at http://127.16.202.2:38089/ using document root <none> and password file <none>
I20260925 00:56:57.253340 1395 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:57.253423 1395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:57.257402 1395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:56:57.259899 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.261729 1395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:56:57.261888 1395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a"
format_stamp: "Formatted at 2026-09-25 00:56:14 on dist-test-slave-rpk1"
server_key: "4b2a3b016faaf7511a8c55fac8897691"
server_key_iv: "e396739908f4da63d9b4cbaa073af770"
server_key_version: "encryptionkey@0"
I20260925 00:56:57.262526 1395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:57.281522 1395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:57.284910 1395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:57.285105 1395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:57.285748 1395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:57.286607 1395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:57.286655 1395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.286695 1395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:57.286723 1395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.295478 1395 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:32827
I20260925 00:56:57.295554 1524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:32827 every 8 connection(s)
I20260925 00:56:57.296530 1395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:56:57.300415 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1395
I20260925 00:56:57.301014 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:38389
--local_ip_for_outbound_sockets=127.16.202.3
--tserver_master_addrs=127.16.202.62:36881
--webserver_port=39563
--webserver_interface=127.16.202.3
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:56:57.301195 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.298889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:53573)
0925 00:56:57.299401 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.299527 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":109,"threads_started":1}
W20260925 00:56:57.301748 1525 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:36881 (0 consecutive failures): Network error: Failed to ping master at 127.16.202.62:36881: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
I20260925 00:56:57.302491 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.302192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:49685)
0925 00:56:57.302339 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.302409 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 00:56:57.303203 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.303005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:43927)
0925 00:56:57.303108 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.303152 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 00:56:57.303363 1525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:56:57.419556 1528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:56:57.419866 1528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:56:57.419912 1528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:56:57.424338 1528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:56:57.424425 1528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:56:57.424491 1528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:56:57.430644 1528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:38389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=39563
--tserver_master_addrs=127.16.202.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:56:57.431772 1528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:56:57.432864 1528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:56:57.442608 1534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:57.442607 1536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:56:57.442619 1533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:56:57.442927 1528 server_base.cc:1034] running on GCE node
I20260925 00:56:57.443527 1528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:56:57.444208 1528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:56:57.445433 1528 hybrid_clock.cc:648] HybridClock initialized: now 1790297817445400 us; error 51 us; skew 500 ppm
Sep 25 00:56:57 dist-test-slave-rpk1 krb5kdc[32541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:56:57.448362 1528 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:56:57.449349 1528 webserver.cc:460] Webserver started at http://127.16.202.3:39563/ using document root <none> and password file <none>
I20260925 00:56:57.449901 1528 fs_manager.cc:359] Metadata directory not provided
I20260925 00:56:57.449959 1528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:56:57.453943 1528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 00:56:57.456347 1543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.457907 1528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:56:57.458039 1528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "ae007bbaca2943a2b30a9e51ebbc5eec"
format_stamp: "Formatted at 2026-09-25 00:56:15 on dist-test-slave-rpk1"
server_key: "9673cc3bb90feffc1bde3c35b3805bbe"
server_key_iv: "6535b470be4030f4b1458dad3f4a6df9"
server_key_version: "encryptionkey@0"
I20260925 00:56:57.458647 1528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:56:57.468210 1528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:56:57.471731 1528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:56:57.471941 1528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:56:57.472630 1528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:56:57.473755 1528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:56:57.473819 1528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.473878 1528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:56:57.473908 1528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:56:57.482702 1528 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:38389
I20260925 00:56:57.482729 1656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:38389 every 8 connection(s)
I20260925 00:56:57.483806 1528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:56:57.487864 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.486031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:55469)
0925 00:56:57.486527 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.486645 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":113,"threads_started":1}
W20260925 00:56:57.488322 1657 heartbeater.cc:644] Failed to heartbeat to 127.16.202.62:36881 (0 consecutive failures): Network error: Failed to ping master at 127.16.202.62:36881: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
I20260925 00:56:57.488646 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1528
I20260925 00:56:57.489038 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.488755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:34489)
0925 00:56:57.488913 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.488971 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 00:56:57.489642 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:57.489432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:51543)
0925 00:56:57.489535 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:57.489584 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:56:57.489837 1657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:56:58.142066 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.141567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:59767)
0925 00:56:58.141869 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:58.141943 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260925 00:56:58.304383 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.303930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:49951)
0925 00:56:58.304201 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:58.304263 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260925 00:56:58.490877 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.490433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:45935)
0925 00:56:58.490714 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:58.490772 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260925 00:56:58.597110 1664 raft_consensus.cc:482] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:56:58.597312 1664 raft_consensus.cc:504] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:56:58.598568 1664 leader_election.cc:288] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389), dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
Sep 25 00:56:58 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.3@KRBTEST.COM
Sep 25 00:56:58 dist-test-slave-rpk1 krb5kdc[32541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.2@KRBTEST.COM
I20260925 00:56:58.613404 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.598804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32827 (local address 127.16.202.1:46665)
0925 00:56:58.599491 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:58.599506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:58.599618 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:58.601347 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:58.601349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:58.601620 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:58.602177 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:58.602199 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.604110 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.604113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:58.605157 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:58.605168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.605907 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.605911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:58.606110 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:58.606749 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:58.606767 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:58.608630 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:58.610878 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:58.610881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:58.610882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:58.611243 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:58.611585 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:58.611588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:58.611590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:58.611746 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:58.612379 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:58.612385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:58.612631 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 00:56:58.612835 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 00:56:58.613223 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":188,"spinlock_wait_cycles":55936,"thread_start_us":86,"threads_started":1}
I20260925 00:56:58.613417 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.599066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:56105 (local address 127.16.202.3:38389)
0925 00:56:58.599558 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:58.599580 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:58.599618 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:56:58.600300 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:58.600306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:58.600657 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:58.600975 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:58.602382 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.603922 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.605282 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.605777 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.605929 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:58.608702 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:58.608710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:58.608730 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:58.608767 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:58.610675 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:58.611362 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:58.611368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:58.611372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:58.611490 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:58.611835 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:58.611838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:58.611840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:58.612266 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:58.612388 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:58.612960 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 00:56:58.613226 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":143,"threads_started":1}
I20260925 00:56:58.613418 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.598997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:46665 (local address 127.16.202.2:32827)
0925 00:56:58.599425 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 00:56:58.599447 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 00:56:58.599599 (+ 152us) server_negotiation.cc:407] Connection header received
0925 00:56:58.600299 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:56:58.600305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:56:58.600707 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:56:58.600953 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:56:58.602372 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.603922 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.605284 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.605774 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.605928 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:58.608822 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:56:58.608828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:56:58.608841 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:56:58.608872 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:56:58.610759 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:58.611360 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:58.611365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:58.611369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:58.611495 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:56:58.611839 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:56:58.611842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:56:58.611843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:56:58.612266 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:56:58.612388 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:56:58.612959 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 00:56:58.613219 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":85,"threads_started":1}
I20260925 00:56:58.613492 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:58.598698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:38389 (local address 127.16.202.1:56105)
0925 00:56:58.598918 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:58.598936 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:56:58.599045 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:56:58.601076 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:56:58.601082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:56:58.601623 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:56:58.602177 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:58.602193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.604075 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.604078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:58.605156 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:56:58.605168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:56:58.605905 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:56:58.605909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:56:58.606116 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:56:58.606799 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:56:58.606810 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:56:58.608419 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:56:58.610837 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:58.610843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:58.610858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:58.611251 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:58.611584 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:56:58.611587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:56:58.611589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:56:58.611744 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:56:58.612381 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:56:58.612387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:56:58.612634 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 00:56:58.612834 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 00:56:58.613301 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":300,"spinlock_wait_cycles":43392}
W20260925 00:56:58.616582 1278 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
W20260925 00:56:58.616753 1280 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
I20260925 00:56:58.616876 1280 leader_election.cc:302] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9; no voters: ae007bbaca2943a2b30a9e51ebbc5eec, dd2e6a9ad7754b5b8070269fbf06bd5a
I20260925 00:56:58.617136 1664 raft_consensus.cc:2726] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:56:59.143321 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:59.142844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:52655)
0925 00:56:59.143167 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:59.143226 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260925 00:56:59.305701 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:59.305219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:38941)
0925 00:56:59.305532 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:59.305602 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260925 00:56:59.492110 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:56:59.491677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:56771)
0925 00:56:59.491950 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 00:56:59.492019 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
I20260925 00:57:00.144786 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:00.144222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:41333)
0925 00:57:00.144563 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:00.144632 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260925 00:57:00.307202 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:00.306598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:57217)
0925 00:57:00.307002 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:00.307093 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260925 00:57:00.428606 1673 raft_consensus.cc:482] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:00.428761 1673 raft_consensus.cc:504] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:57:00.429152 1673 leader_election.cc:288] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389), dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
W20260925 00:57:00.429908 1280 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
W20260925 00:57:00.430042 1278 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
I20260925 00:57:00.430122 1278 leader_election.cc:302] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9; no voters: ae007bbaca2943a2b30a9e51ebbc5eec, dd2e6a9ad7754b5b8070269fbf06bd5a
I20260925 00:57:00.430258 1673 raft_consensus.cc:2726] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:57:00.493456 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:00.492985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:51199)
0925 00:57:00.493290 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:00.493355 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
I20260925 00:57:01.146355 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:01.145874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:36329)
0925 00:57:01.146180 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:01.146240 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260925 00:57:01.308629 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:01.308075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:34697)
0925 00:57:01.308453 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:01.308525 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260925 00:57:01.494959 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:01.494524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:35733)
0925 00:57:01.494798 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:01.494861 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260925 00:57:02.148110 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:02.147602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:56847)
0925 00:57:02.147945 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:02.148011 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":77,"threads_started":1}
I20260925 00:57:02.309945 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:02.309510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:35957)
0925 00:57:02.309782 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:02.309844 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260925 00:57:02.496268 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:02.495820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:57147)
0925 00:57:02.496108 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:02.496172 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
I20260925 00:57:02.834069 1681 raft_consensus.cc:482] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:02.834203 1681 raft_consensus.cc:504] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:57:02.834630 1681 leader_election.cc:288] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389), dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
W20260925 00:57:02.835402 1280 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
W20260925 00:57:02.835551 1278 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
I20260925 00:57:02.835642 1278 leader_election.cc:302] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9; no voters: ae007bbaca2943a2b30a9e51ebbc5eec, dd2e6a9ad7754b5b8070269fbf06bd5a
I20260925 00:57:02.835789 1681 raft_consensus.cc:2726] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:57:03.149338 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:03.148915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:49617)
0925 00:57:03.149164 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:03.149222 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260925 00:57:03.311224 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:03.310709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:50025)
0925 00:57:03.311046 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:03.311113 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260925 00:57:03.497679 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:03.497185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:37263)
0925 00:57:03.497469 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:03.497534 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260925 00:57:04.150650 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:04.150204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:55653)
0925 00:57:04.150477 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:04.150549 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260925 00:57:04.312587 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:04.312084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:53531)
0925 00:57:04.312430 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:04.312488 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260925 00:57:04.498881 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:04.498435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:60715)
0925 00:57:04.498727 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:04.498785 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
I20260925 00:57:05.151947 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:05.151521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:43589)
0925 00:57:05.151791 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:05.151848 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260925 00:57:05.313828 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:05.313394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:42911)
0925 00:57:05.313666 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:05.313730 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260925 00:57:05.500164 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:05.499694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:33849)
0925 00:57:05.499991 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:05.500061 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260925 00:57:06.153385 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:06.152904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:47021)
0925 00:57:06.153220 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:06.153287 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260925 00:57:06.315037 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:06.314606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:57839)
0925 00:57:06.314872 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:06.314937 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260925 00:57:06.342945 1693 raft_consensus.cc:482] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:06.343082 1693 raft_consensus.cc:504] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae007bbaca2943a2b30a9e51ebbc5eec" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 38389 } } peers { permanent_uuid: "dd2e6a9ad7754b5b8070269fbf06bd5a" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32827 } } peers { permanent_uuid: "6363f966d1dc41c78977b86385f7e8c9" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 39183 } }
I20260925 00:57:06.343532 1693 leader_election.cc:288] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389), dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827)
W20260925 00:57:06.344240 1278 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd2e6a9ad7754b5b8070269fbf06bd5a (127.16.202.2:32827): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
W20260925 00:57:06.344413 1280 leader_election.cc:341] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae007bbaca2943a2b30a9e51ebbc5eec (127.16.202.3:38389): Not found: Tablet not found: 269894963a594b2ea73eca2fb74b43ed
I20260925 00:57:06.344484 1280 leader_election.cc:302] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6363f966d1dc41c78977b86385f7e8c9; no voters: ae007bbaca2943a2b30a9e51ebbc5eec, dd2e6a9ad7754b5b8070269fbf06bd5a
I20260925 00:57:06.344622 1693 raft_consensus.cc:2726] T 269894963a594b2ea73eca2fb74b43ed P 6363f966d1dc41c78977b86385f7e8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:57:06.501416 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:06.501012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:47591)
0925 00:57:06.501261 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:06.501320 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260925 00:57:07.154753 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:07.154314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:33761)
0925 00:57:07.154588 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:07.154651 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260925 00:57:07.316403 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:07.315968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:51937)
0925 00:57:07.316243 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:07.316302 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260925 00:57:07.502719 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:07.502254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:34789)
0925 00:57:07.502543 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:07.502613 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
I20260925 00:57:08.156122 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:08.155688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:43547)
0925 00:57:08.155943 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:08.156005 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260925 00:57:08.317620 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:08.317198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:37435)
0925 00:57:08.317452 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:08.317517 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260925 00:57:08.503968 1700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:08.503543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:54489)
0925 00:57:08.503782 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:08.503844 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260925 00:57:09.157393 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:09.156980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:52995)
0925 00:57:09.157233 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:09.157294 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260925 00:57:09.318881 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:09.318414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:48465)
0925 00:57:09.318716 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:09.318778 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260925 00:57:09.505136 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:09.504715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:50637)
0925 00:57:09.504966 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:09.505028 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260925 00:57:10.158707 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:10.158186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:33451)
0925 00:57:10.158473 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:10.158573 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260925 00:57:10.320103 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:10.319644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:41819)
0925 00:57:10.319917 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:10.319975 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260925 00:57:10.506299 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:10.505831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:41003)
0925 00:57:10.506126 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:10.506185 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260925 00:57:11.160004 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:11.159588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:34919)
0925 00:57:11.159834 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:11.159905 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260925 00:57:11.321458 1708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:11.320998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:36897)
0925 00:57:11.321286 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:11.321352 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260925 00:57:11.507694 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:11.507216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.3:47129)
0925 00:57:11.507529 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:11.507596 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":81,"threads_started":1}
I20260925 00:57:12.161581 1710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:12.161042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.1:48085)
0925 00:57:12.161396 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:12.161474 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260925 00:57:12.322892 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:12.322415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36881 (local address 127.16.202.2:55975)
0925 00:57:12.322731 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:12.322790 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.62:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20260925 00:57:12.497920 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 1221
I20260925 00:57:12.503149 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 1395
I20260925 00:57:12.507992 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 1528
2026-09-25T00:57:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59259 ms)
[----------] 30 tests from SecurityITest (241318 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:12 dist-test-slave-rpk1 krb5kdc[1714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:12 dist-test-slave-rpk1 krb5kdc[1714](info): set up 2 sockets
Sep 25 00:57:12 dist-test-slave-rpk1 krb5kdc[1714](info): commencing operation
krb5kdc: starting...
W20260925 00:57:13.484541 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:13 dist-test-slave-rpk1 krb5kdc[1714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:13.607133 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:44029
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:13.719511 1727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:13.719841 1727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:13.719904 1727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:13.724159 1727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:13.724248 1727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:13.724277 1727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:13.724298 1727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:13.724318 1727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:13.729975 1727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:44029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:13.731118 1727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:13.732172 1727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:13.741626 1732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:13.741680 1735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:13.741667 1733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:13.741739 1727 server_base.cc:1034] running on GCE node
I20260925 00:57:13.742623 1727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:13.743831 1727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:13.745003 1727 hybrid_clock.cc:648] HybridClock initialized: now 1790297833744986 us; error 37 us; skew 500 ppm
Sep 25 00:57:13 dist-test-slave-rpk1 krb5kdc[1714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:13.747889 1727 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:13.748943 1727 webserver.cc:460] Webserver started at http://127.16.202.62:36489/ using document root <none> and password file <none>
I20260925 00:57:13.749512 1727 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:13.749577 1727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:13.749776 1727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:13.751896 1727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "0d92e9c40e39482e911b3b2916bec436"
format_stamp: "Formatted at 2026-09-25 00:57:13 on dist-test-slave-rpk1"
server_key: "5bc83c26c441de76a1b4b938b56021bc"
server_key_iv: "0002a0286d1f7f0fb6883ea267a34939"
server_key_version: "encryptionkey@0"
I20260925 00:57:13.752497 1727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "0d92e9c40e39482e911b3b2916bec436"
format_stamp: "Formatted at 2026-09-25 00:57:13 on dist-test-slave-rpk1"
server_key: "5bc83c26c441de76a1b4b938b56021bc"
server_key_iv: "0002a0286d1f7f0fb6883ea267a34939"
server_key_version: "encryptionkey@0"
I20260925 00:57:13.756536 1727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:57:13.759320 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:13.760656 1727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:57:13.760859 1727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "0d92e9c40e39482e911b3b2916bec436"
format_stamp: "Formatted at 2026-09-25 00:57:13 on dist-test-slave-rpk1"
server_key: "5bc83c26c441de76a1b4b938b56021bc"
server_key_iv: "0002a0286d1f7f0fb6883ea267a34939"
server_key_version: "encryptionkey@0"
I20260925 00:57:13.760972 1727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:13.789539 1727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:13.793059 1727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:13.793272 1727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:13.800268 1727 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:44029
I20260925 00:57:13.800282 1794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:44029 every 8 connection(s)
I20260925 00:57:13.801311 1727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:13.804087 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1727
I20260925 00:57:13.804227 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:13.804287 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:13.804545 17192 external_mini_cluster.cc:1428] Setting key 71e2160cee6bf45c8b9e93129f4a0b96
I20260925 00:57:13.810954 1795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436: Bootstrap starting.
Sep 25 00:57:13 dist-test-slave-rpk1 krb5kdc[1714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:13.813385 1795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:13.814105 1795 log.cc:824] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:13.816152 1795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436: No bootstrap required, opened a new log
I20260925 00:57:13.818607 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:13.806082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.16.202.62:44029)
0925 00:57:13.806580 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:13.806595 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:13.806631 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:57:13.807294 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:13.807361 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:13.807717 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:13.808049 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:13.808918 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:13.809946 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:13.810778 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:13.811293 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:13.811454 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:13.813951 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:13.813956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:13.813972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:13.814003 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:13.815678 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:13.816105 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:13.816110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:13.816115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:13.816225 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:13.816554 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:13.816556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:13.816558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:13.817042 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:13.817197 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:13.817523 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 00:57:13.817851 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":154,"threads_started":1}
I20260925 00:57:13.820125 1795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d92e9c40e39482e911b3b2916bec436" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44029 } }
I20260925 00:57:13.820405 1795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:13.820464 1795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d92e9c40e39482e911b3b2916bec436, State: Initialized, Role: FOLLOWER
I20260925 00:57:13.820917 1795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d92e9c40e39482e911b3b2916bec436" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44029 } }
I20260925 00:57:13.821055 1795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:13.821102 1795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:13.821172 1795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:13.822063 1795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d92e9c40e39482e911b3b2916bec436" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44029 } }
I20260925 00:57:13.822391 1795 leader_election.cc:302] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d92e9c40e39482e911b3b2916bec436; no voters:
I20260925 00:57:13.822722 1795 leader_election.cc:288] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:13.822824 1800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:13.823025 1800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [term 1 LEADER]: Becoming Leader. State: Replica: 0d92e9c40e39482e911b3b2916bec436, State: Running, Role: LEADER
I20260925 00:57:13.823357 1800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d92e9c40e39482e911b3b2916bec436" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44029 } }
I20260925 00:57:13.824105 1795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:13.825165 1801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d92e9c40e39482e911b3b2916bec436. Latest consensus state: current_term: 1 leader_uuid: "0d92e9c40e39482e911b3b2916bec436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d92e9c40e39482e911b3b2916bec436" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44029 } } }
I20260925 00:57:13.825209 1802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d92e9c40e39482e911b3b2916bec436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d92e9c40e39482e911b3b2916bec436" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 44029 } } }
I20260925 00:57:13.825404 1801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:13.825517 1802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:13.826527 1810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:13.829869 1810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:13.835620 1810 catalog_manager.cc:1347] Generated new cluster ID: 1d18457896f34dd08bbf79792f67b0e1
I20260925 00:57:13.835709 1810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:13.865814 1810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:13.866888 1810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:13.872033 1810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436: Generated new TSK 0
I20260925 00:57:13.872815 1810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:57:13.907891 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44029
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:14.023030 1821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:14.023391 1821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:14.023443 1821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:14.027916 1821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:14.028005 1821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:14.028087 1821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:57:14.033691 1821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:14.034763 1821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:14.035753 1821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:14.045224 1826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:14.045395 1821 server_base.cc:1034] running on GCE node
W20260925 00:57:14.045207 1829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:14.045202 1827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:14.046003 1821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:14.046669 1821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:14.047894 1821 hybrid_clock.cc:648] HybridClock initialized: now 1790297834047879 us; error 31 us; skew 500 ppm
Sep 25 00:57:14 dist-test-slave-rpk1 krb5kdc[1714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:14.050848 1821 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:57:14.051755 1821 webserver.cc:460] Webserver started at http://127.16.202.1:39433/ using document root <none> and password file <none>
I20260925 00:57:14.052320 1821 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:14.052382 1821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:14.052621 1821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:14.054946 1821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "c23d7231491d4c20a6410b84c7248203"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "af7b3fc704822a4b887952974e3bb5a3"
server_key_iv: "421d7f866f3dcb26be95f4dd52f8781a"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.055570 1821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "c23d7231491d4c20a6410b84c7248203"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "af7b3fc704822a4b887952974e3bb5a3"
server_key_iv: "421d7f866f3dcb26be95f4dd52f8781a"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.059737 1821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:14.062567 1836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.063987 1821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:57:14.064119 1821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "c23d7231491d4c20a6410b84c7248203"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "af7b3fc704822a4b887952974e3bb5a3"
server_key_iv: "421d7f866f3dcb26be95f4dd52f8781a"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.064236 1821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:14.084635 1821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:14.088078 1821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:14.088275 1821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:14.088951 1821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:14.089838 1821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:14.089887 1821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.089944 1821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:14.089974 1821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.098780 1821 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:33297
I20260925 00:57:14.098802 1949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:33297 every 8 connection(s)
I20260925 00:57:14.099700 1821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:57:14.104803 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1821
I20260925 00:57:14.104924 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:57:14.105196 17192 external_mini_cluster.cc:1428] Setting key 855115ed2ea80061a25378bd64119f89
Sep 25 00:57:14 dist-test-slave-rpk1 krb5kdc[1714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:14.114627 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.101527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:44475 (local address 127.16.202.62:44029)
0925 00:57:14.101673 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.101678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.102576 (+ 898us) server_negotiation.cc:407] Connection header received
0925 00:57:14.103876 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.103878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.103923 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.104003 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:14.105317 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.105930 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.107304 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.107628 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.107763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.110532 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:14.110536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:14.110539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:14.110573 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:14.112318 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.112936 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.112940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.112942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.113040 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.113375 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.113378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.113380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.113640 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:14.113744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.114300 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.114455 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:57:14.115381 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.101961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:44029 (local address 127.16.202.1:44475)
0925 00:57:14.102431 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.102474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.103686 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.104155 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.104162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.104563 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:14.105148 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.105166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.106056 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.106059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.107168 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.107178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.107813 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.107817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.108003 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.108622 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:14.108639 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:14.110315 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:14.112475 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:14.112482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:14.112500 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:14.112836 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:14.113137 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:14.113139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:14.113141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:14.113280 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:14.113739 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:14.113744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:14.114034 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.114196 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.114473 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260925 00:57:14.116971 1950 heartbeater.cc:347] Connected to a master server at 127.16.202.62:44029
I20260925 00:57:14.117205 1950 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:57:14.117748 1950 heartbeater.cc:511] Master 127.16.202.62:44029 requested a full tablet report, sending...
I20260925 00:57:14.119344 1759 ts_manager.cc:194] Registered new tserver with Master: c23d7231491d4c20a6410b84c7248203 (127.16.202.1:33297)
I20260925 00:57:14.120908 1759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:44475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:57:14.135645 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44029
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:14.246196 1955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:14.246513 1955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:14.246577 1955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:14.251070 1955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:14.251140 1955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:14.251220 1955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:57:14.257212 1955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:14.258354 1955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:14.259375 1955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:14.268862 1963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:14.268862 1961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:14.268945 1960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:14.269224 1955 server_base.cc:1034] running on GCE node
I20260925 00:57:14.269663 1955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:14.270325 1955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:14.271508 1955 hybrid_clock.cc:648] HybridClock initialized: now 1790297834271490 us; error 29 us; skew 500 ppm
Sep 25 00:57:14 dist-test-slave-rpk1 krb5kdc[1714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:14.274308 1955 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:57:14.275240 1955 webserver.cc:460] Webserver started at http://127.16.202.2:35011/ using document root <none> and password file <none>
I20260925 00:57:14.275797 1955 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:14.275863 1955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:14.276058 1955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:14.278143 1955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "aaf8df70e47b43249fe1b3e5a1e14be6"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "748a97bb06322a028266ca7a2970ceaa"
server_key_iv: "ddbf0a6620446dcaa84afd37a1dca0ff"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.278673 1955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "aaf8df70e47b43249fe1b3e5a1e14be6"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "748a97bb06322a028266ca7a2970ceaa"
server_key_iv: "ddbf0a6620446dcaa84afd37a1dca0ff"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.282563 1955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:14.285346 1970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.286664 1955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:14.286799 1955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "aaf8df70e47b43249fe1b3e5a1e14be6"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "748a97bb06322a028266ca7a2970ceaa"
server_key_iv: "ddbf0a6620446dcaa84afd37a1dca0ff"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.286908 1955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:14.315858 1955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:14.319080 1955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:14.319259 1955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:14.319854 1955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:14.320727 1955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:14.320788 1955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.320849 1955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:14.320875 1955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.329586 1955 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:38465
I20260925 00:57:14.329609 2083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:38465 every 8 connection(s)
I20260925 00:57:14.330473 1955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:57:14.332653 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 1955
I20260925 00:57:14.332767 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:57:14.333086 17192 external_mini_cluster.cc:1428] Setting key 5ea0bd912c180028a84ce050035ae480
Sep 25 00:57:14 dist-test-slave-rpk1 krb5kdc[1714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:57:14.344969 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.332210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:49439 (local address 127.16.202.62:44029)
0925 00:57:14.332373 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.332378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.333313 (+ 935us) server_negotiation.cc:407] Connection header received
0925 00:57:14.334715 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.334717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.334761 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.334826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:14.336326 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.337142 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.338413 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.338761 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.338869 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.341308 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:14.341312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:14.341315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:14.341344 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:14.342871 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.343390 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.343393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.343394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.343515 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.343813 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.343815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.343816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.344003 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:14.344077 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.344601 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.344791 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:57:14.345786 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.332591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:44029 (local address 127.16.202.2:49439)
0925 00:57:14.333147 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.333207 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.334482 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.335034 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.335043 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.335504 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:14.336163 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.336183 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.337283 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.337286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.338279 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.338290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.338903 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.338906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.339069 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.339694 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:14.339710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:14.341134 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:14.342978 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:14.342983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:14.342997 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:14.343266 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:14.343604 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:14.343607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:14.343608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:14.343744 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:14.344094 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:14.344100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:14.344379 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.344559 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.344899 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":128,"threads_started":1}
I20260925 00:57:14.347123 2084 heartbeater.cc:347] Connected to a master server at 127.16.202.62:44029
I20260925 00:57:14.347360 2084 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:14.347918 2084 heartbeater.cc:511] Master 127.16.202.62:44029 requested a full tablet report, sending...
I20260925 00:57:14.349148 1759 ts_manager.cc:194] Registered new tserver with Master: aaf8df70e47b43249fe1b3e5a1e14be6 (127.16.202.2:38465)
I20260925 00:57:14.349815 1759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:49439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:57:14.363842 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44029
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:14.477183 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:14.477479 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:14.477524 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:14.481789 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:14.481863 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:14.481927 2089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:57:14.487923 2089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:14.489084 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:14.490217 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:14.500353 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:14.500340 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:14.500336 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:14.500600 2089 server_base.cc:1034] running on GCE node
I20260925 00:57:14.501248 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:14.501959 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:14.503139 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790297834503109 us; error 44 us; skew 500 ppm
Sep 25 00:57:14 dist-test-slave-rpk1 krb5kdc[1714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:14.506162 2089 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:57:14.507244 2089 webserver.cc:460] Webserver started at http://127.16.202.3:43913/ using document root <none> and password file <none>
I20260925 00:57:14.507835 2089 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:14.507913 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:14.508131 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:14.510514 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "1ac9caf856a34846815467a325d26dd8"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "d4ff071d1f5c513302eed0ee094fef53"
server_key_iv: "30ca218c05ca877f5967d5921e0d4bba"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.511163 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "1ac9caf856a34846815467a325d26dd8"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "d4ff071d1f5c513302eed0ee094fef53"
server_key_iv: "30ca218c05ca877f5967d5921e0d4bba"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.515255 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 00:57:14.517988 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.519371 2089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:57:14.519503 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "1ac9caf856a34846815467a325d26dd8"
format_stamp: "Formatted at 2026-09-25 00:57:14 on dist-test-slave-rpk1"
server_key: "d4ff071d1f5c513302eed0ee094fef53"
server_key_iv: "30ca218c05ca877f5967d5921e0d4bba"
server_key_version: "encryptionkey@0"
I20260925 00:57:14.519621 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:14.535790 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:14.539534 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:14.539752 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:14.540424 2089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:14.541392 2089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:14.541450 2089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.541509 2089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:14.541540 2089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:14.550268 2089 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:37787
I20260925 00:57:14.550294 2217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:37787 every 8 connection(s)
I20260925 00:57:14.551175 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:57:14.560837 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2089
I20260925 00:57:14.560964 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:57:14.561214 17192 external_mini_cluster.cc:1428] Setting key fed52d3735767b1928c4fac42365c579
Sep 25 00:57:14 dist-test-slave-rpk1 krb5kdc[1714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:14.565599 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.553123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:48161 (local address 127.16.202.62:44029)
0925 00:57:14.553253 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.553256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.554278 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 00:57:14.555724 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.555728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.555792 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.555868 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:14.557190 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.557728 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.558900 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.559127 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.559216 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.561970 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:14.561975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:14.561977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:14.562008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:14.563474 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.563958 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.563960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.563962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.564031 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.564381 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.564383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.564384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.564601 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:14.564706 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.565313 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.565456 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:57:14.566511 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.553554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:44029 (local address 127.16.202.3:48161)
0925 00:57:14.554103 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.554152 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.555493 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.556070 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.556077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.556473 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:14.557043 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.557059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.557851 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.557854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.558774 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.558783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.559277 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.559281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.559467 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.560066 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:14.560084 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:14.561784 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:14.563586 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:14.563592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:14.563605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:14.563877 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:14.564112 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:14.564114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:14.564116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:14.564281 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:14.564731 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:14.564738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:14.565054 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.565233 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.565623 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":126,"threads_started":1}
I20260925 00:57:14.567736 2218 heartbeater.cc:347] Connected to a master server at 127.16.202.62:44029
I20260925 00:57:14.567972 2218 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:14.568535 2218 heartbeater.cc:511] Master 127.16.202.62:44029 requested a full tablet report, sending...
I20260925 00:57:14.569630 1759 ts_manager.cc:194] Registered new tserver with Master: 1ac9caf856a34846815467a325d26dd8 (127.16.202.3:37787)
I20260925 00:57:14.570218 1759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:48161
I20260925 00:57:14.575555 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:57:14.584949 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.577772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.16.202.62:44029)
0925 00:57:14.577913 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.577917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.578094 (+ 177us) server_negotiation.cc:407] Connection header received
0925 00:57:14.578173 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.578175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.578220 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.578294 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:14.578977 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.579501 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.580363 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.580612 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.580723 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.582170 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:14.582173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:14.582174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:14.582198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:14.583373 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.583752 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.583755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.583756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.583828 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:14.584074 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:14.584077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:14.584079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:14.584304 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:14.584421 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.584654 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.584797 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 00:57:14.587954 1759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:57:14.591245 1759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:57:14.606032 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.600150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.16.202.1:33297)
0925 00:57:14.601097 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.601112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.601138 (+ 26us) server_negotiation.cc:407] Connection header received
0925 00:57:14.601217 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.601223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.601464 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.601954 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:14.602122 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.603298 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.604793 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.605375 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.605471 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.605631 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.605762 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.605872 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":86,"threads_started":1}
I20260925 00:57:14.606027 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.600021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:33297 (local address 127.0.0.1:59966)
0925 00:57:14.600404 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.600460 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.600588 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.601640 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.601646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.601668 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:14.601801 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.601808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.603436 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.603440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.604650 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.604658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.605487 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.605490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.605594 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.605630 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.605734 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.605872 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":54,"thread_start_us":102,"threads_started":1}
I20260925 00:57:14.606328 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.600826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:38465 (local address 127.0.0.1:33700)
0925 00:57:14.601454 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.601467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.601555 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.601948 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.601951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.601965 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:14.602065 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.602071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.603105 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.603108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.604502 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.604513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.605814 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.605818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.606019 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.606057 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.606122 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.606196 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":81,"thread_start_us":110,"threads_started":1}
I20260925 00:57:14.606699 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.600204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:37787 (local address 127.0.0.1:43650)
0925 00:57:14.601696 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.601709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.601801 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.602206 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.602208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.602224 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:14.602316 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.602322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.603573 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.603576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.604642 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.604653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.605814 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.605818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.606014 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.606053 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.606514 (+ 461us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.606584 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"mutex_wait_us":857,"thread_start_us":84,"threads_started":1}
I20260925 00:57:14.606735 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.601079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.16.202.2:38465)
0925 00:57:14.601540 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.601552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.601573 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:57:14.601649 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.601654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.601842 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.601978 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:14.602152 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.602987 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.604632 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.605506 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.605612 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.605752 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.606503 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.606600 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":125,"threads_started":1}
I20260925 00:57:14.607195 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.600601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.16.202.3:37787)
0925 00:57:14.601324 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.601334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.601806 (+ 472us) server_negotiation.cc:407] Connection header received
0925 00:57:14.601912 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.601917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.602089 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.602232 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:14.602404 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.603458 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.604797 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.605568 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.605650 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.605756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.606958 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.607058 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":192,"threads_started":1}
I20260925 00:57:14.609609 2152 tablet_service.cc:1467] Processing CreateTablet for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f (DEFAULT_TABLE table=test-table [id=f3922e0108a345ba9a5894682c0a11d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:14.609833 1884 tablet_service.cc:1467] Processing CreateTablet for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f (DEFAULT_TABLE table=test-table [id=f3922e0108a345ba9a5894682c0a11d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:14.609836 2018 tablet_service.cc:1467] Processing CreateTablet for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f (DEFAULT_TABLE table=test-table [id=f3922e0108a345ba9a5894682c0a11d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:14.610565 2152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:14.610857 1884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:14.610870 2018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:14.617283 2235 tablet_bootstrap.cc:492] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Bootstrap starting.
I20260925 00:57:14.619076 2236 tablet_bootstrap.cc:492] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: Bootstrap starting.
I20260925 00:57:14.619573 2235 tablet_bootstrap.cc:654] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:14.620713 2235 log.cc:824] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:14.621016 2236 tablet_bootstrap.cc:654] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:14.621081 2237 tablet_bootstrap.cc:492] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Bootstrap starting.
I20260925 00:57:14.621870 2236 log.cc:824] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:14.622951 2235 tablet_bootstrap.cc:492] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: No bootstrap required, opened a new log
I20260925 00:57:14.623145 2235 ts_tablet_manager.cc:1397] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:57:14.623754 2237 tablet_bootstrap.cc:654] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:14.623896 2236 tablet_bootstrap.cc:492] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: No bootstrap required, opened a new log
I20260925 00:57:14.624086 2236 ts_tablet_manager.cc:1397] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:57:14.624601 2237 log.cc:824] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:14.626649 2235 raft_consensus.cc:348] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.626906 2235 raft_consensus.cc:374] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:14.626961 2235 raft_consensus.cc:729] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac9caf856a34846815467a325d26dd8, State: Initialized, Role: FOLLOWER
I20260925 00:57:14.626979 2237 tablet_bootstrap.cc:492] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: No bootstrap required, opened a new log
I20260925 00:57:14.627169 2237 ts_tablet_manager.cc:1397] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 00:57:14.627413 2235 consensus_queue.cc:260] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.628062 2218 heartbeater.cc:503] Master 127.16.202.62:44029 was elected leader, sending a full tablet report...
I20260925 00:57:14.628458 2235 ts_tablet_manager.cc:1428] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:57:14.628654 2236 raft_consensus.cc:348] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.629033 2236 raft_consensus.cc:374] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:14.629083 2236 raft_consensus.cc:729] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23d7231491d4c20a6410b84c7248203, State: Initialized, Role: FOLLOWER
I20260925 00:57:14.629558 2236 consensus_queue.cc:260] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.630265 1950 heartbeater.cc:503] Master 127.16.202.62:44029 was elected leader, sending a full tablet report...
I20260925 00:57:14.631078 2237 raft_consensus.cc:348] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.631711 2236 ts_tablet_manager.cc:1428] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260925 00:57:14.631371 2237 raft_consensus.cc:374] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:14.632200 2237 raft_consensus.cc:729] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf8df70e47b43249fe1b3e5a1e14be6, State: Initialized, Role: FOLLOWER
I20260925 00:57:14.632758 2237 consensus_queue.cc:260] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.633665 2237 ts_tablet_manager.cc:1428] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:57:14.634006 2084 heartbeater.cc:503] Master 127.16.202.62:44029 was elected leader, sending a full tablet report...
I20260925 00:57:14.788389 2242 raft_consensus.cc:482] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:14.788650 2242 raft_consensus.cc:504] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.789918 2242 leader_election.cc:288] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf8df70e47b43249fe1b3e5a1e14be6 (127.16.202.2:38465), 1ac9caf856a34846815467a325d26dd8 (127.16.202.3:37787)
I20260925 00:57:14.794227 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.790401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:47023 (local address 127.16.202.2:38465)
0925 00:57:14.790727 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.790731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.790745 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:57:14.790953 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.790956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.791005 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.791168 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:14.791481 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.792029 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.793311 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.793695 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.793789 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.793843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.794024 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.794095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260925 00:57:14.794247 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.790474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:37227 (local address 127.16.202.3:37787)
0925 00:57:14.790602 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.790607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.790702 (+ 95us) server_negotiation.cc:407] Connection header received
0925 00:57:14.790975 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.790978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.791121 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.791187 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:14.791650 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.792269 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.793326 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.793730 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.793840 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.793889 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.794058 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.794128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:14.794277 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.790241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:37787 (local address 127.16.202.1:37227)
0925 00:57:14.790618 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.790632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.790805 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.791341 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.791344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.791361 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:14.791493 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.791500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.792394 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.792398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.793208 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.793217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.793851 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.793854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.793943 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.793960 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.794030 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.794094 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":18,"spinlock_wait_cycles":5504,"thread_start_us":129,"threads_started":1}
I20260925 00:57:14.794286 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.790250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:38465 (local address 127.16.202.1:47023)
0925 00:57:14.790664 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.790678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:14.790802 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:14.791187 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:14.791190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:14.791208 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:14.791346 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.791353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.792166 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.792170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.793182 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:14.793192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.793809 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.793813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:14.793904 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.793928 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:57:14.793990 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 00:57:14.794063 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":4,"spinlock_wait_cycles":26624,"thread_start_us":119,"threads_started":1}
I20260925 00:57:14.794867 2172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ca4ff2ee104967b30ea1ab43ef9b0f" candidate_uuid: "c23d7231491d4c20a6410b84c7248203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac9caf856a34846815467a325d26dd8" is_pre_election: true
I20260925 00:57:14.795028 2038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ca4ff2ee104967b30ea1ab43ef9b0f" candidate_uuid: "c23d7231491d4c20a6410b84c7248203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" is_pre_election: true
I20260925 00:57:14.795202 2172 raft_consensus.cc:2454] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23d7231491d4c20a6410b84c7248203 in term 0.
I20260925 00:57:14.795327 2038 raft_consensus.cc:2454] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23d7231491d4c20a6410b84c7248203 in term 0.
I20260925 00:57:14.795678 1838 leader_election.cc:302] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac9caf856a34846815467a325d26dd8, c23d7231491d4c20a6410b84c7248203; no voters:
I20260925 00:57:14.795966 2242 raft_consensus.cc:2781] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:57:14.796051 2242 raft_consensus.cc:482] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:57:14.796100 2242 raft_consensus.cc:3037] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:14.797215 2242 raft_consensus.cc:504] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
I20260925 00:57:14.797570 2242 leader_election.cc:288] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [CANDIDATE]: Term 1 election: Requested vote from peers aaf8df70e47b43249fe1b3e5a1e14be6 (127.16.202.2:38465), 1ac9caf856a34846815467a325d26dd8 (127.16.202.3:37787)
I20260925 00:57:14.798015 2172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ca4ff2ee104967b30ea1ab43ef9b0f" candidate_uuid: "c23d7231491d4c20a6410b84c7248203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac9caf856a34846815467a325d26dd8"
I20260925 00:57:14.797986 2038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ca4ff2ee104967b30ea1ab43ef9b0f" candidate_uuid: "c23d7231491d4c20a6410b84c7248203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6"
I20260925 00:57:14.798130 2172 raft_consensus.cc:3037] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:14.798137 2038 raft_consensus.cc:3037] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:14.799153 2172 raft_consensus.cc:2454] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23d7231491d4c20a6410b84c7248203 in term 1.
I20260925 00:57:14.799222 2038 raft_consensus.cc:2454] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23d7231491d4c20a6410b84c7248203 in term 1.
I20260925 00:57:14.799461 1838 leader_election.cc:302] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac9caf856a34846815467a325d26dd8, c23d7231491d4c20a6410b84c7248203; no voters:
I20260925 00:57:14.799628 2242 raft_consensus.cc:2781] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:14.799829 2242 raft_consensus.cc:686] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 1 LEADER]: Becoming Leader. State: Replica: c23d7231491d4c20a6410b84c7248203, State: Running, Role: LEADER
I20260925 00:57:14.800194 2242 consensus_queue.cc:237] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } }
W20260925 00:57:14.802240 2219 tablet.cc:2369] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:14.803584 1757 catalog_manager.cc:5495] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 reported cstate change: term changed from 0 to 1, leader changed from <none> to c23d7231491d4c20a6410b84c7248203 (127.16.202.1). New cstate: current_term: 1 leader_uuid: "c23d7231491d4c20a6410b84c7248203" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23d7231491d4c20a6410b84c7248203" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 33297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 } health_report { overall_health: UNKNOWN } } }
W20260925 00:57:14.831619 2085 tablet.cc:2369] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:14.845865 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.842428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.16.202.1:33297)
0925 00:57:14.842560 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:14.842564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:14.842577 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:57:14.842735 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:14.842737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:14.842794 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:14.842883 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:14.843189 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.843699 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.844576 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:14.844831 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:14.844911 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:14.845153 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:14.845577 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:14.845663 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:14.845706 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 00:57:14.845753 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 00:57:14.851064 1951 tablet.cc:2369] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:14.851936 2172 raft_consensus.cc:1261] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 1 FOLLOWER]: Refusing update from remote peer c23d7231491d4c20a6410b84c7248203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:14.851936 2038 raft_consensus.cc:1261] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 1 FOLLOWER]: Refusing update from remote peer c23d7231491d4c20a6410b84c7248203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:14.852689 2246 consensus_queue.cc:1035] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf8df70e47b43249fe1b3e5a1e14be6" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:14.852864 2242 consensus_queue.cc:1035] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac9caf856a34846815467a325d26dd8" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 37787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:14.860329 2249 mvcc.cc:205] Tried to move back new op lower bound from 7333059931545624576 to 7333059931344683008. Current Snapshot: MvccSnapshot[applied={T|T < 7333059931545624576}]
I20260925 00:57:14.861320 2250 mvcc.cc:205] Tried to move back new op lower bound from 7333059931545624576 to 7333059931344683008. Current Snapshot: MvccSnapshot[applied={T|T < 7333059931545624576}]
I20260925 00:57:14.869961 1757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:14.870124 1757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:14.872495 1757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d92e9c40e39482e911b3b2916bec436: Sending DeleteTablet for 3 replicas of tablet b5ca4ff2ee104967b30ea1ab43ef9b0f
I20260925 00:57:14.873248 1884 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:14 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:59966
I20260925 00:57:14.873420 2018 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:14 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:33700
I20260925 00:57:14.873770 2259 tablet_replica.cc:330] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203: stopping tablet replica
I20260925 00:57:14.873804 2152 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:14 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:43650
I20260925 00:57:14.874110 2259 raft_consensus.cc:2229] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:57:14.874289 2261 tablet_replica.cc:330] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: stopping tablet replica
I20260925 00:57:14.874549 2259 raft_consensus.cc:2258] T b5ca4ff2ee104967b30ea1ab43ef9b0f P c23d7231491d4c20a6410b84c7248203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:14.874672 2261 raft_consensus.cc:2229] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:14.874929 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 1821
I20260925 00:57:14.875110 2261 raft_consensus.cc:2258] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:14.876616 2261 ts_tablet_manager.cc:1905] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:14.876888 2260 tablet_replica.cc:330] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: stopping tablet replica
I20260925 00:57:14.877159 2260 raft_consensus.cc:2229] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:14.877439 2260 raft_consensus.cc:2258] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:14.878719 2260 ts_tablet_manager.cc:1905] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:14.879748 2261 ts_tablet_manager.cc:1918] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:14.879845 2261 log.cc:1196] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/b5ca4ff2ee104967b30ea1ab43ef9b0f
I20260925 00:57:14.880156 2261 ts_tablet_manager.cc:1939] T b5ca4ff2ee104967b30ea1ab43ef9b0f P 1ac9caf856a34846815467a325d26dd8: Deleting consensus metadata
I20260925 00:57:14.880862 2260 ts_tablet_manager.cc:1918] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:14.880947 2260 log.cc:1196] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/b5ca4ff2ee104967b30ea1ab43ef9b0f
I20260925 00:57:14.881186 2260 ts_tablet_manager.cc:1939] T b5ca4ff2ee104967b30ea1ab43ef9b0f P aaf8df70e47b43249fe1b3e5a1e14be6: Deleting consensus metadata
I20260925 00:57:14.881443 1744 catalog_manager.cc:4841] TS 1ac9caf856a34846815467a325d26dd8 (127.16.202.3:37787): tablet b5ca4ff2ee104967b30ea1ab43ef9b0f (table test-table [id=f3922e0108a345ba9a5894682c0a11d3]) successfully deleted
I20260925 00:57:14.882156 1743 catalog_manager.cc:4841] TS aaf8df70e47b43249fe1b3e5a1e14be6 (127.16.202.2:38465): tablet b5ca4ff2ee104967b30ea1ab43ef9b0f (table test-table [id=f3922e0108a345ba9a5894682c0a11d3]) successfully deleted
W20260925 00:57:14.882853 1743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:33297 (error 108)
I20260925 00:57:14.883011 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 1955
I20260925 00:57:14.885603 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:14.885311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:33297 (local address 127.0.0.1:59986)
0925 00:57:14.885450 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:14.885521 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:33297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:57:14.885803 1743 catalog_manager.cc:4584] TS c23d7231491d4c20a6410b84c7248203 (127.16.202.1:33297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5ca4ff2ee104967b30ea1ab43ef9b0f: Network error: Client connection negotiation failed: client connection to 127.16.202.1:33297: connect: Connection refused (error 111)
I20260925 00:57:14.888808 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2089
I20260925 00:57:14.894459 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 1727
2026-09-25T00:57:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2386 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T00:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:14Z Disabled control of system clock
I20260925 00:57:14.923466 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34501
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:57:15.033725 2266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:15.034037 2266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:15.034073 2266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:15.038153 2266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:15.038242 2266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:15.038265 2266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:15.038285 2266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:15.038305 2266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:15.043418 2266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:34501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:15.044597 2266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:15.046087 2266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:15.055181 2271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.055166 2274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:15.055241 2266 server_base.cc:1034] running on GCE node
W20260925 00:57:15.055140 2272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:15.056099 2266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:15.057335 2266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:15.058542 2266 hybrid_clock.cc:648] HybridClock initialized: now 1790297835058519 us; error 40 us; skew 500 ppm
I20260925 00:57:15.060735 2266 webserver.cc:460] Webserver started at http://127.16.202.62:42097/ using document root <none> and password file <none>
I20260925 00:57:15.061343 2266 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:15.061416 2266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:15.061643 2266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:15.063822 2266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "d9bbb890aeea4af78255f1a1105f9e59"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "a846aad8dcba769e997c3ef6def4c818"
server_key_iv: "6664cfa892fc28160caa22a62de4c5cf"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.064390 2266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "d9bbb890aeea4af78255f1a1105f9e59"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "a846aad8dcba769e997c3ef6def4c818"
server_key_iv: "6664cfa892fc28160caa22a62de4c5cf"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.068300 2266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 00:57:15.070983 2280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.072228 2266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:57:15.072346 2266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "d9bbb890aeea4af78255f1a1105f9e59"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "a846aad8dcba769e997c3ef6def4c818"
server_key_iv: "6664cfa892fc28160caa22a62de4c5cf"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.072448 2266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:15.095104 2266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:15.095867 2266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:15.096069 2266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:15.103006 2266 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:34501
I20260925 00:57:15.103041 2332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:34501 every 8 connection(s)
I20260925 00:57:15.104014 2266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:15.107234 2333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:15.109746 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2266
I20260925 00:57:15.109869 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:15.110163 17192 external_mini_cluster.cc:1428] Setting key 826c80f2f6905cb4b35614dcf4dee232
I20260925 00:57:15.114323 2333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59: Bootstrap starting.
I20260925 00:57:15.116951 2333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:15.117875 2333 log.cc:824] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:15.120164 2333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59: No bootstrap required, opened a new log
I20260925 00:57:15.120173 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.111705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.16.202.62:34501)
0925 00:57:15.112340 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.112360 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.112397 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:57:15.113297 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.113318 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.113763 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.114068 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:15.114457 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.115443 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.116608 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.117179 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.117300 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.118179 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:15.118185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:15.118199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:15.118763 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:15.118828 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:15.118850 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:15.118932 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.119143 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.119446 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260925 00:57:15.124257 2333 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9bbb890aeea4af78255f1a1105f9e59" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34501 } }
I20260925 00:57:15.124507 2333 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:15.124543 2333 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9bbb890aeea4af78255f1a1105f9e59, State: Initialized, Role: FOLLOWER
I20260925 00:57:15.124990 2333 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9bbb890aeea4af78255f1a1105f9e59" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34501 } }
I20260925 00:57:15.125128 2333 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:15.125209 2333 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:15.125310 2333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:15.126315 2333 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9bbb890aeea4af78255f1a1105f9e59" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34501 } }
I20260925 00:57:15.126645 2333 leader_election.cc:302] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9bbb890aeea4af78255f1a1105f9e59; no voters:
I20260925 00:57:15.126991 2333 leader_election.cc:288] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:15.127122 2338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:15.127343 2338 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [term 1 LEADER]: Becoming Leader. State: Replica: d9bbb890aeea4af78255f1a1105f9e59, State: Running, Role: LEADER
I20260925 00:57:15.127679 2338 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9bbb890aeea4af78255f1a1105f9e59" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34501 } }
I20260925 00:57:15.128291 2333 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:15.129500 2339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9bbb890aeea4af78255f1a1105f9e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9bbb890aeea4af78255f1a1105f9e59" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34501 } } }
I20260925 00:57:15.129532 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9bbb890aeea4af78255f1a1105f9e59. Latest consensus state: current_term: 1 leader_uuid: "d9bbb890aeea4af78255f1a1105f9e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9bbb890aeea4af78255f1a1105f9e59" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 34501 } } }
I20260925 00:57:15.129629 2339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:15.129640 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:15.130033 2347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:15.133213 2347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:15.139251 2347 catalog_manager.cc:1347] Generated new cluster ID: a98367c479874fbab4cce8bb2fa1062f
I20260925 00:57:15.139359 2347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:15.151899 2347 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:15.153301 2347 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:15.165174 2347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59: Generated new TSK 0
I20260925 00:57:15.166129 2347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:15.172305 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34501
--builtin_ntp_servers=127.16.202.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:57:15.288615 2357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:15.288932 2357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:15.288992 2357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:15.293183 2357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:15.293263 2357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:15.293342 2357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:57:15.298425 2357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:15.299578 2357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:15.300997 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:15.310792 2363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.310887 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.310824 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:15.311113 2357 server_base.cc:1034] running on GCE node
I20260925 00:57:15.311673 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:15.312337 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:15.313529 2357 hybrid_clock.cc:648] HybridClock initialized: now 1790297835313489 us; error 58 us; skew 500 ppm
I20260925 00:57:15.315562 2357 webserver.cc:460] Webserver started at http://127.16.202.1:39243/ using document root <none> and password file <none>
I20260925 00:57:15.316134 2357 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:15.316196 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:15.316411 2357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:15.318571 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "c5b56ec3d57c41d1b7cf57be8d28a164"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "9ee946f4e47cccba6830e6c591c1ff69"
server_key_iv: "139b741d485a1556bbf7003121fb1d06"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.319092 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "c5b56ec3d57c41d1b7cf57be8d28a164"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "9ee946f4e47cccba6830e6c591c1ff69"
server_key_iv: "139b741d485a1556bbf7003121fb1d06"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.322932 2357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:15.325615 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.327031 2357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:15.327170 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "c5b56ec3d57c41d1b7cf57be8d28a164"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "9ee946f4e47cccba6830e6c591c1ff69"
server_key_iv: "139b741d485a1556bbf7003121fb1d06"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.327287 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:15.348812 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:15.349588 2357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:15.349782 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:15.350404 2357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:15.351411 2357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:15.351460 2357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.351516 2357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:15.351545 2357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.360251 2357 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:44345
I20260925 00:57:15.360291 2484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:44345 every 8 connection(s)
I20260925 00:57:15.361215 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:57:15.369318 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2357
I20260925 00:57:15.369428 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:57:15.369735 17192 external_mini_cluster.cc:1428] Setting key b4c36cdece56e690421accefbbebd543
I20260925 00:57:15.370539 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.363138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:37233 (local address 127.16.202.62:34501)
0925 00:57:15.363264 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.363268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.364359 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 00:57:15.365866 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.365871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.365928 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.366002 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:15.366920 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.367458 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.368580 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.368873 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.368966 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.370081 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:15.370085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:15.370087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:15.370101 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:15.370122 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:15.370130 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:15.370186 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.370331 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.370394 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:57:15.371479 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.363551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34501 (local address 127.16.202.1:37233)
0925 00:57:15.364208 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.364249 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.365631 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.366186 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.366193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.366281 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:15.366778 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.366790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.367572 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.367574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.368457 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.368466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.369017 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.369021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.369176 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.369817 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:15.369819 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:15.369869 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:15.369937 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:15.370195 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:15.370204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:15.370210 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.370320 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.370683 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260925 00:57:15.371740 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34501
--builtin_ntp_servers=127.16.202.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:57:15.372843 2485 heartbeater.cc:347] Connected to a master server at 127.16.202.62:34501
I20260925 00:57:15.373136 2485 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:15.373699 2485 heartbeater.cc:511] Master 127.16.202.62:34501 requested a full tablet report, sending...
I20260925 00:57:15.375222 2295 ts_manager.cc:194] Registered new tserver with Master: c5b56ec3d57c41d1b7cf57be8d28a164 (127.16.202.1:44345)
I20260925 00:57:15.376421 2295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.1:37233
W20260925 00:57:15.487978 2488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:15.488278 2488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:15.488318 2488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:15.494673 2488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:15.494784 2488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:15.494853 2488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:57:15.500942 2488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:15.502087 2488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:15.503571 2488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:15.514266 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.514266 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.514487 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:15.514487 2488 server_base.cc:1034] running on GCE node
I20260925 00:57:15.515164 2488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:15.515919 2488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:15.517081 2488 hybrid_clock.cc:648] HybridClock initialized: now 1790297835517066 us; error 25 us; skew 500 ppm
I20260925 00:57:15.519181 2488 webserver.cc:460] Webserver started at http://127.16.202.2:40171/ using document root <none> and password file <none>
I20260925 00:57:15.519747 2488 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:15.519809 2488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:15.520025 2488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:15.522181 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "e321a5b8e64e4cf1bab1c43256aa3788"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "c72bf0639e33cc88e9fb6907ac44425c"
server_key_iv: "bf3e81a5487202c6de6db49c1aefbd2d"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.522727 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "e321a5b8e64e4cf1bab1c43256aa3788"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "c72bf0639e33cc88e9fb6907ac44425c"
server_key_iv: "bf3e81a5487202c6de6db49c1aefbd2d"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.526592 2488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:15.529397 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.530930 2488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:15.531046 2488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "e321a5b8e64e4cf1bab1c43256aa3788"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "c72bf0639e33cc88e9fb6907ac44425c"
server_key_iv: "bf3e81a5487202c6de6db49c1aefbd2d"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.531178 2488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:15.553525 2488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:15.554313 2488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:15.554606 2488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:15.555335 2488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:15.556348 2488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:15.556394 2488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.556438 2488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:15.556470 2488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.565609 2488 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:33699
I20260925 00:57:15.565654 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:33699 every 8 connection(s)
I20260925 00:57:15.566537 2488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:57:15.568840 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2488
I20260925 00:57:15.568969 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:57:15.569267 17192 external_mini_cluster.cc:1428] Setting key ed01da49b419e6a2c3d1432d866e6876
I20260925 00:57:15.571367 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34501
--builtin_ntp_servers=127.16.202.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:57:15.577128 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.568362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:39113 (local address 127.16.202.62:34501)
0925 00:57:15.568502 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.568506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.569847 (+ 1341us) server_negotiation.cc:407] Connection header received
0925 00:57:15.571485 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.571489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.571549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.571636 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:15.572875 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.573674 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.574857 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.575185 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.575297 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.576571 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:15.576575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:15.576578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:15.576595 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:15.576621 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:15.576632 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:15.576748 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.576892 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.576963 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:57:15.577983 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.568895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34501 (local address 127.16.202.2:39113)
0925 00:57:15.569701 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.569745 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.571226 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.571838 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.571848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.571962 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:15.572647 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.572667 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.573803 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.573806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.574716 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.574725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.575334 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.575337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.575587 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.576234 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:15.576235 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:15.576308 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:15.576399 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:15.576749 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:15.576757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:15.576763 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.576868 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.577121 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260925 00:57:15.579088 2616 heartbeater.cc:347] Connected to a master server at 127.16.202.62:34501
I20260925 00:57:15.579329 2616 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:15.579865 2616 heartbeater.cc:511] Master 127.16.202.62:34501 requested a full tablet report, sending...
I20260925 00:57:15.580930 2295 ts_manager.cc:194] Registered new tserver with Master: e321a5b8e64e4cf1bab1c43256aa3788 (127.16.202.2:33699)
I20260925 00:57:15.581527 2295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.2:39113
W20260925 00:57:15.686923 2619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:15.687224 2619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:15.687280 2619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:15.691414 2619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:15.691481 2619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:15.691561 2619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:57:15.696463 2619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:15.697688 2619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:15.699105 2619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:57:15.709002 2619 server_base.cc:1034] running on GCE node
W20260925 00:57:15.708909 2625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.708920 2627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:15.708909 2624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:15.709616 2619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:15.710317 2619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:15.711516 2619 hybrid_clock.cc:648] HybridClock initialized: now 1790297835711495 us; error 35 us; skew 500 ppm
I20260925 00:57:15.713492 2619 webserver.cc:460] Webserver started at http://127.16.202.3:43767/ using document root <none> and password file <none>
I20260925 00:57:15.714075 2619 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:15.714145 2619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:15.714370 2619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:15.716495 2619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "f599aa8baed2457b9172ee916fb1b9fa"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "99b0436ba0f950312c814217f1e9682f"
server_key_iv: "91503332c5adc768b741b46e06dbf1f4"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.717099 2619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "f599aa8baed2457b9172ee916fb1b9fa"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "99b0436ba0f950312c814217f1e9682f"
server_key_iv: "91503332c5adc768b741b46e06dbf1f4"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.720852 2619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:15.723348 2633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.724612 2619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:57:15.724767 2619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "f599aa8baed2457b9172ee916fb1b9fa"
format_stamp: "Formatted at 2026-09-25 00:57:15 on dist-test-slave-rpk1"
server_key: "99b0436ba0f950312c814217f1e9682f"
server_key_iv: "91503332c5adc768b741b46e06dbf1f4"
server_key_version: "encryptionkey@0"
I20260925 00:57:15.724879 2619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:15.748720 2619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:15.749454 2619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:15.749748 2619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:15.750404 2619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:15.751433 2619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:15.751484 2619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.751539 2619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:15.751566 2619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:15.760716 2619 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:41449
I20260925 00:57:15.760747 2746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:41449 every 8 connection(s)
I20260925 00:57:15.761682 2619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:57:15.768419 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2619
I20260925 00:57:15.768515 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:57:15.768793 17192 external_mini_cluster.cc:1428] Setting key b39a69418ad37a1b06ab683ddbc34205
I20260925 00:57:15.771543 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.763727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:53489 (local address 127.16.202.62:34501)
0925 00:57:15.763858 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.763862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.765067 (+ 1205us) server_negotiation.cc:407] Connection header received
0925 00:57:15.766501 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.766505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.766567 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.766645 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:15.767614 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.768182 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.769589 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.769816 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.769916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.771066 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:15.771070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:15.771073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:15.771093 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:15.771117 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:15.771125 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:15.771182 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.771332 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.771398 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:57:15.772466 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.764124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:34501 (local address 127.16.202.3:53489)
0925 00:57:15.764877 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.764925 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.766258 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.766838 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.766846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.766942 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:15.767462 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.767476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.768313 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.768316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.769465 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.769476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.769962 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.769966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.770140 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.770792 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:15.770794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:15.770847 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:15.770923 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:15.771193 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:15.771203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:15.771208 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.771322 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.771691 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":123,"threads_started":1}
I20260925 00:57:15.773551 2747 heartbeater.cc:347] Connected to a master server at 127.16.202.62:34501
I20260925 00:57:15.773758 2747 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:15.774240 2747 heartbeater.cc:511] Master 127.16.202.62:34501 requested a full tablet report, sending...
I20260925 00:57:15.775183 2295 ts_manager.cc:194] Registered new tserver with Master: f599aa8baed2457b9172ee916fb1b9fa (127.16.202.3:41449)
I20260925 00:57:15.775758 2295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.3:53489
I20260925 00:57:15.782696 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:57:15.788544 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.784796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.16.202.62:34501)
0925 00:57:15.784957 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.784962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.785095 (+ 133us) server_negotiation.cc:407] Connection header received
0925 00:57:15.785229 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.785232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.785279 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.785352 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:15.785656 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.786208 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.787026 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.787254 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.787339 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.788123 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:15.788127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:15.788130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:15.788144 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:15.788173 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:15.788181 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:15.788239 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.788338 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.788399 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:15.791464 2295 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:57:15.794628 2295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:57:15.809291 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.803579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.16.202.2:33699)
0925 00:57:15.804970 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.804981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.805003 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:57:15.805091 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.805096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.805289 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.805427 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:15.805726 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.806466 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.807887 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.808617 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.808749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.808979 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.809031 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.809127 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":143,"threads_started":1}
I20260925 00:57:15.809290 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.803433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33699 (local address 127.0.0.1:43476)
0925 00:57:15.804271 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.804325 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.804466 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.805434 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.805440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.805462 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:15.805622 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.805627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.806571 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.806575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.807750 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.807761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.808744 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.808747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.808855 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.808892 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.808995 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.809124 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":2,"spinlock_wait_cycles":39808,"thread_start_us":98,"threads_started":1}
I20260925 00:57:15.809893 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.803706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44345 (local address 127.0.0.1:51748)
0925 00:57:15.804695 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.804707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.804780 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.805483 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.805486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.805519 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:15.805654 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.805661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.807145 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.807148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.808153 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.808170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.809533 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.809536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.809609 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.809626 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.809675 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.809751 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":7,"spinlock_wait_cycles":4352,"thread_start_us":81,"threads_started":1}
I20260925 00:57:15.810276 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.803520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:41449 (local address 127.0.0.1:55500)
0925 00:57:15.804270 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.804325 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.804466 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.805483 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.805486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.805516 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:15.805664 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.805669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.807111 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.807114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.808248 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.808263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.809476 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.809481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.809573 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.809596 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.809665 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.810182 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":535,"spinlock_wait_cycles":19072,"thread_start_us":94,"threads_started":1}
I20260925 00:57:15.810366 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.804093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.16.202.1:44345)
0925 00:57:15.804738 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.804750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.804775 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:57:15.804945 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.804952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.805182 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.805332 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:15.805779 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.806991 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.808282 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.809123 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.809224 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.809964 (+ 740us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.810107 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.810204 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":130,"threads_started":1}
I20260925 00:57:15.810485 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.804253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.16.202.3:41449)
0925 00:57:15.804874 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.804887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.804908 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:57:15.804969 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.804972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.805182 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.805331 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:15.805779 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.806991 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.808367 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.809120 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.809224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.809408 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.809789 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.809904 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":159,"threads_started":1}
I20260925 00:57:15.812985 2681 tablet_service.cc:1467] Processing CreateTablet for tablet ab46568a3afe49b0b0ac5270dddfd0ba (DEFAULT_TABLE table=test-table [id=679d26668d9f40aa9c2383310bdcf4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:15.812870 2550 tablet_service.cc:1467] Processing CreateTablet for tablet ab46568a3afe49b0b0ac5270dddfd0ba (DEFAULT_TABLE table=test-table [id=679d26668d9f40aa9c2383310bdcf4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:15.813153 2419 tablet_service.cc:1467] Processing CreateTablet for tablet ab46568a3afe49b0b0ac5270dddfd0ba (DEFAULT_TABLE table=test-table [id=679d26668d9f40aa9c2383310bdcf4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:15.814100 2681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab46568a3afe49b0b0ac5270dddfd0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:15.814100 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab46568a3afe49b0b0ac5270dddfd0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:15.814107 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab46568a3afe49b0b0ac5270dddfd0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:15.821601 2765 tablet_bootstrap.cc:492] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Bootstrap starting.
I20260925 00:57:15.822400 2764 tablet_bootstrap.cc:492] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: Bootstrap starting.
I20260925 00:57:15.822747 2766 tablet_bootstrap.cc:492] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Bootstrap starting.
I20260925 00:57:15.824419 2765 tablet_bootstrap.cc:654] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:15.824419 2764 tablet_bootstrap.cc:654] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:15.824505 2766 tablet_bootstrap.cc:654] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:15.825320 2766 log.cc:824] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:15.825345 2764 log.cc:824] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:15.825393 2765 log.cc:824] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:15.827440 2766 tablet_bootstrap.cc:492] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: No bootstrap required, opened a new log
I20260925 00:57:15.827440 2765 tablet_bootstrap.cc:492] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: No bootstrap required, opened a new log
I20260925 00:57:15.827653 2765 ts_tablet_manager.cc:1397] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:57:15.827653 2766 ts_tablet_manager.cc:1397] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 00:57:15.827744 2764 tablet_bootstrap.cc:492] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: No bootstrap required, opened a new log
I20260925 00:57:15.827965 2764 ts_tablet_manager.cc:1397] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:57:15.831527 2764 raft_consensus.cc:348] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.831823 2764 raft_consensus.cc:374] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:15.831899 2764 raft_consensus.cc:729] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b56ec3d57c41d1b7cf57be8d28a164, State: Initialized, Role: FOLLOWER
I20260925 00:57:15.832075 2766 raft_consensus.cc:348] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.832075 2765 raft_consensus.cc:348] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.832342 2766 raft_consensus.cc:374] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:15.832342 2765 raft_consensus.cc:374] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:15.832398 2766 raft_consensus.cc:729] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e321a5b8e64e4cf1bab1c43256aa3788, State: Initialized, Role: FOLLOWER
I20260925 00:57:15.832405 2765 raft_consensus.cc:729] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f599aa8baed2457b9172ee916fb1b9fa, State: Initialized, Role: FOLLOWER
I20260925 00:57:15.832368 2764 consensus_queue.cc:260] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.832899 2765 consensus_queue.cc:260] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.832906 2766 consensus_queue.cc:260] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.833454 2764 ts_tablet_manager.cc:1428] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 00:57:15.833904 2766 ts_tablet_manager.cc:1428] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:57:15.833896 2765 ts_tablet_manager.cc:1428] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 00:57:15.834097 2485 heartbeater.cc:503] Master 127.16.202.62:34501 was elected leader, sending a full tablet report...
I20260925 00:57:15.834229 2747 heartbeater.cc:503] Master 127.16.202.62:34501 was elected leader, sending a full tablet report...
I20260925 00:57:15.834229 2616 heartbeater.cc:503] Master 127.16.202.62:34501 was elected leader, sending a full tablet report...
W20260925 00:57:15.862519 2486 tablet.cc:2369] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:15.970476 2771 raft_consensus.cc:482] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:15.970738 2771 raft_consensus.cc:504] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.971993 2771 leader_election.cc:288] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e321a5b8e64e4cf1bab1c43256aa3788 (127.16.202.2:33699), c5b56ec3d57c41d1b7cf57be8d28a164 (127.16.202.1:44345)
I20260925 00:57:15.976768 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.972207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33699 (local address 127.16.202.3:51517)
0925 00:57:15.972398 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.972413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.972550 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.972847 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.972850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.972863 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:15.972973 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.972981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.974073 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.974077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.975556 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.975569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.976269 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.976273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.976379 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.976431 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.976505 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.976588 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":92}
I20260925 00:57:15.976799 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.972316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:51517 (local address 127.16.202.2:33699)
0925 00:57:15.972425 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.972430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.972595 (+ 165us) server_negotiation.cc:407] Connection header received
0925 00:57:15.972660 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.972662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.972738 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.972835 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:15.973092 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.973888 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.975737 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.976146 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.976260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.976309 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.976544 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.976635 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:57:15.976784 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.972273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44345 (local address 127.16.202.3:47441)
0925 00:57:15.972628 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:15.972640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:15.972758 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:15.973002 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:15.973004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:15.973019 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:15.973129 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.973134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.974047 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.974051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.975556 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:15.975569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.976371 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.976374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:15.976486 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.976505 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:57:15.976599 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 00:57:15.976659 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":114,"thread_start_us":136,"threads_started":1}
I20260925 00:57:15.976824 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.972429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:47441 (local address 127.16.202.1:44345)
0925 00:57:15.972569 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.972574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.972739 (+ 165us) server_negotiation.cc:407] Connection header received
0925 00:57:15.972845 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.972848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.972896 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.972968 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:15.973221 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.973939 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.975720 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.976139 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:15.976245 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:15.976364 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:15.976623 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 00:57:15.976721 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:57:15.977469 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab46568a3afe49b0b0ac5270dddfd0ba" candidate_uuid: "f599aa8baed2457b9172ee916fb1b9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" is_pre_election: true
I20260925 00:57:15.977725 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab46568a3afe49b0b0ac5270dddfd0ba" candidate_uuid: "f599aa8baed2457b9172ee916fb1b9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" is_pre_election: true
I20260925 00:57:15.977800 2570 raft_consensus.cc:2454] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f599aa8baed2457b9172ee916fb1b9fa in term 0.
I20260925 00:57:15.977994 2439 raft_consensus.cc:2454] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f599aa8baed2457b9172ee916fb1b9fa in term 0.
I20260925 00:57:15.978343 2635 leader_election.cc:302] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e321a5b8e64e4cf1bab1c43256aa3788, f599aa8baed2457b9172ee916fb1b9fa; no voters:
I20260925 00:57:15.978616 2771 raft_consensus.cc:2781] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:57:15.978700 2771 raft_consensus.cc:482] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:57:15.978749 2771 raft_consensus.cc:3037] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:15.979727 2771 raft_consensus.cc:504] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.980067 2771 leader_election.cc:288] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [CANDIDATE]: Term 1 election: Requested vote from peers e321a5b8e64e4cf1bab1c43256aa3788 (127.16.202.2:33699), c5b56ec3d57c41d1b7cf57be8d28a164 (127.16.202.1:44345)
I20260925 00:57:15.980428 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab46568a3afe49b0b0ac5270dddfd0ba" candidate_uuid: "f599aa8baed2457b9172ee916fb1b9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164"
I20260925 00:57:15.980438 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab46568a3afe49b0b0ac5270dddfd0ba" candidate_uuid: "f599aa8baed2457b9172ee916fb1b9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e321a5b8e64e4cf1bab1c43256aa3788"
I20260925 00:57:15.980563 2570 raft_consensus.cc:3037] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:15.980592 2439 raft_consensus.cc:3037] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:15.981881 2570 raft_consensus.cc:2454] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f599aa8baed2457b9172ee916fb1b9fa in term 1.
I20260925 00:57:15.981889 2439 raft_consensus.cc:2454] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f599aa8baed2457b9172ee916fb1b9fa in term 1.
I20260925 00:57:15.982213 2635 leader_election.cc:302] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c5b56ec3d57c41d1b7cf57be8d28a164, e321a5b8e64e4cf1bab1c43256aa3788, f599aa8baed2457b9172ee916fb1b9fa; no voters:
I20260925 00:57:15.982409 2771 raft_consensus.cc:2781] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:15.982636 2771 raft_consensus.cc:686] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 1 LEADER]: Becoming Leader. State: Replica: f599aa8baed2457b9172ee916fb1b9fa, State: Running, Role: LEADER
I20260925 00:57:15.982957 2771 consensus_queue.cc:237] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } }
I20260925 00:57:15.986377 2295 catalog_manager.cc:5495] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa reported cstate change: term changed from 0 to 1, leader changed from <none> to f599aa8baed2457b9172ee916fb1b9fa (127.16.202.3). New cstate: current_term: 1 leader_uuid: "f599aa8baed2457b9172ee916fb1b9fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f599aa8baed2457b9172ee916fb1b9fa" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 41449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 } health_report { overall_health: UNKNOWN } } }
I20260925 00:57:16.001832 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:15.998339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.16.202.3:41449)
0925 00:57:15.998473 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:15.998476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:15.998489 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:57:15.998565 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:15.998567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:15.998617 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:15.998703 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:15.998878 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:15.999372 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:16.000289 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:16.000534 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:16.000623 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:16.000993 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:16.001419 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:16.001559 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:16.001610 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 00:57:16.001664 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:57:16.007848 2570 raft_consensus.cc:1261] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 1 FOLLOWER]: Refusing update from remote peer f599aa8baed2457b9172ee916fb1b9fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:16.007848 2439 raft_consensus.cc:1261] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164 [term 1 FOLLOWER]: Refusing update from remote peer f599aa8baed2457b9172ee916fb1b9fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:16.008598 2774 consensus_queue.cc:1035] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e321a5b8e64e4cf1bab1c43256aa3788" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:16.008805 2771 consensus_queue.cc:1035] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b56ec3d57c41d1b7cf57be8d28a164" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:57:16.013213 2748 tablet.cc:2369] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:16.016397 2777 mvcc.cc:205] Tried to move back new op lower bound from 7333059936280875008 to 7333059936189075456. Current Snapshot: MvccSnapshot[applied={T|T < 7333059936280875008}]
I20260925 00:57:16.020618 2779 mvcc.cc:205] Tried to move back new op lower bound from 7333059936280875008 to 7333059936189075456. Current Snapshot: MvccSnapshot[applied={T|T < 7333059936280875008 or (T in {7333059936280875008})}]
I20260925 00:57:16.024454 2295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:16.024627 2295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:16.026998 2295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9bbb890aeea4af78255f1a1105f9e59: Sending DeleteTablet for 3 replicas of tablet ab46568a3afe49b0b0ac5270dddfd0ba
I20260925 00:57:16.027704 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet ab46568a3afe49b0b0ac5270dddfd0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:16 UTC) from {username='slave'} at 127.0.0.1:43476
I20260925 00:57:16.027737 2681 tablet_service.cc:1514] Processing DeleteTablet for tablet ab46568a3afe49b0b0ac5270dddfd0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:16 UTC) from {username='slave'} at 127.0.0.1:55500
I20260925 00:57:16.028398 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet ab46568a3afe49b0b0ac5270dddfd0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:16 UTC) from {username='slave'} at 127.0.0.1:51748
I20260925 00:57:16.028494 2788 tablet_replica.cc:330] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: stopping tablet replica
I20260925 00:57:16.028853 2788 raft_consensus.cc:2229] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:16.029100 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2357
I20260925 00:57:16.029177 2788 raft_consensus.cc:2258] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:16.029317 2789 tablet_replica.cc:330] T ab46568a3afe49b0b0ac5270dddfd0ba P c5b56ec3d57c41d1b7cf57be8d28a164: stopping tablet replica
I20260925 00:57:16.030115 2787 tablet_replica.cc:330] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: stopping tablet replica
I20260925 00:57:16.030390 2787 raft_consensus.cc:2229] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:57:16.030766 2787 raft_consensus.cc:2258] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:16.032269 2787 ts_tablet_manager.cc:1905] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:16.033806 2788 ts_tablet_manager.cc:1905] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:16.036567 2788 ts_tablet_manager.cc:1918] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 00:57:16.036592 2284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:44345 (error 108)
I20260925 00:57:16.036700 2788 log.cc:1196] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/ab46568a3afe49b0b0ac5270dddfd0ba
I20260925 00:57:16.037025 2788 ts_tablet_manager.cc:1939] T ab46568a3afe49b0b0ac5270dddfd0ba P e321a5b8e64e4cf1bab1c43256aa3788: Deleting consensus metadata
I20260925 00:57:16.037802 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:16.037503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44345 (local address 127.0.0.1:51750)
0925 00:57:16.037634 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:16.037706 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 00:57:16.037827 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2488
I20260925 00:57:16.040530 2787 ts_tablet_manager.cc:1918] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 00:57:16.040498 2284 catalog_manager.cc:4584] TS c5b56ec3d57c41d1b7cf57be8d28a164 (127.16.202.1:44345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab46568a3afe49b0b0ac5270dddfd0ba: Network error: Client connection negotiation failed: client connection to 127.16.202.1:44345: connect: Connection refused (error 111)
I20260925 00:57:16.040633 2787 log.cc:1196] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/ab46568a3afe49b0b0ac5270dddfd0ba
I20260925 00:57:16.040990 2787 ts_tablet_manager.cc:1939] T ab46568a3afe49b0b0ac5270dddfd0ba P f599aa8baed2457b9172ee916fb1b9fa: Deleting consensus metadata
I20260925 00:57:16.042119 2283 catalog_manager.cc:4841] TS f599aa8baed2457b9172ee916fb1b9fa (127.16.202.3:41449): tablet ab46568a3afe49b0b0ac5270dddfd0ba (table test-table [id=679d26668d9f40aa9c2383310bdcf4bc]) successfully deleted
I20260925 00:57:16.045717 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2619
I20260925 00:57:16.047255 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:16.045922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:33699 (local address 127.0.0.1:43478)
0925 00:57:16.047118 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:16.047172 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.2:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1108}
I20260925 00:57:16.050812 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2266
2026-09-25T00:57:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1155 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:16 dist-test-slave-rpk1 krb5kdc[2793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:16 dist-test-slave-rpk1 krb5kdc[2793](info): set up 2 sockets
Sep 25 00:57:16 dist-test-slave-rpk1 krb5kdc[2793](info): commencing operation
krb5kdc: starting...
W20260925 00:57:17.020532 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:17.139688 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38515
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:17.251721 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:17.252058 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:17.252118 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:17.258399 2806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:17.258496 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:17.258526 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:17.258553 2806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:17.258590 2806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:17.264930 2806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:17.266111 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:17.267194 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:17.276305 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:17.276402 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:17.276477 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:17.276852 2806 server_base.cc:1034] running on GCE node
I20260925 00:57:17.277467 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:17.278648 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:17.279843 2806 hybrid_clock.cc:648] HybridClock initialized: now 1790297837279813 us; error 45 us; skew 500 ppm
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:17.282820 2806 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:17.283821 2806 webserver.cc:460] Webserver started at http://127.16.202.62:34879/ using document root <none> and password file <none>
I20260925 00:57:17.284385 2806 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:17.284462 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:17.284713 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:17.286881 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "4f0ba794dfee464688b4085c87eec089"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "94d60d04b7e0027094371696206dc27c"
server_key_iv: "b72ce1d8d87d7c79feed215230fabaa8"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.287482 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "4f0ba794dfee464688b4085c87eec089"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "94d60d04b7e0027094371696206dc27c"
server_key_iv: "b72ce1d8d87d7c79feed215230fabaa8"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.291504 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:17.294167 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.295593 2806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:57:17.295740 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "4f0ba794dfee464688b4085c87eec089"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "94d60d04b7e0027094371696206dc27c"
server_key_iv: "b72ce1d8d87d7c79feed215230fabaa8"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.295858 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:17.319422 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:17.322701 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:17.322894 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:17.330171 2806 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:38515
I20260925 00:57:17.330250 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:38515 every 8 connection(s)
I20260925 00:57:17.331199 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:17.334296 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:17.336457 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2806
I20260925 00:57:17.336593 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:17.336916 17192 external_mini_cluster.cc:1428] Setting key befc272e9dca285abe1d3cbc0a47e856
I20260925 00:57:17.340946 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089: Bootstrap starting.
I20260925 00:57:17.343294 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:17.344107 2874 log.cc:824] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:17.346360 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089: No bootstrap required, opened a new log
I20260925 00:57:17.350276 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0ba794dfee464688b4085c87eec089" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38515 } }
I20260925 00:57:17.350638 2874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:17.350713 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f0ba794dfee464688b4085c87eec089, State: Initialized, Role: FOLLOWER
I20260925 00:57:17.351186 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0ba794dfee464688b4085c87eec089" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38515 } }
I20260925 00:57:17.351318 2874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:17.351382 2874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:17.351490 2874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:17.351524 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:17.338487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.16.202.62:38515)
0925 00:57:17.338961 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:17.338977 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:17.339015 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:57:17.339939 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:17.339961 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:17.340337 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:17.340719 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:17.341578 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.342592 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.343478 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.343991 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.344146 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:17.346800 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:17.346805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:17.346820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:17.346849 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:17.348457 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:17.348947 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:17.348952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:17.348957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:17.349088 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:17.349368 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:17.349371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:17.349372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:17.349800 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:17.350001 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:17.350304 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 00:57:17.350614 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
I20260925 00:57:17.352479 2874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0ba794dfee464688b4085c87eec089" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38515 } }
I20260925 00:57:17.352864 2874 leader_election.cc:302] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f0ba794dfee464688b4085c87eec089; no voters:
I20260925 00:57:17.353204 2874 leader_election.cc:288] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:17.353307 2879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:17.353462 2879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [term 1 LEADER]: Becoming Leader. State: Replica: 4f0ba794dfee464688b4085c87eec089, State: Running, Role: LEADER
I20260925 00:57:17.353837 2879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0ba794dfee464688b4085c87eec089" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38515 } }
I20260925 00:57:17.354610 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:17.355156 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f0ba794dfee464688b4085c87eec089. Latest consensus state: current_term: 1 leader_uuid: "4f0ba794dfee464688b4085c87eec089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0ba794dfee464688b4085c87eec089" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38515 } } }
I20260925 00:57:17.355373 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:17.355546 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f0ba794dfee464688b4085c87eec089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0ba794dfee464688b4085c87eec089" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38515 } } }
I20260925 00:57:17.355683 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:17.355868 2886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:17.359400 2886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:17.365694 2886 catalog_manager.cc:1347] Generated new cluster ID: 222e635a13f04d069b175a1482f38173
I20260925 00:57:17.365826 2886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:17.396389 2886 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:17.397867 2886 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:17.409505 2886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089: Generated new TSK 0
I20260925 00:57:17.410251 2886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:57:17.444756 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38515
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:17.581429 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:17.581709 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:17.581770 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:17.585949 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:17.586025 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:17.586108 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:57:17.591606 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:17.592768 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:17.593801 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:17.605371 2905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:17.605448 2908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:17.605445 2900 server_base.cc:1034] running on GCE node
W20260925 00:57:17.605625 2906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:17.606118 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:17.606801 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:17.607976 2900 hybrid_clock.cc:648] HybridClock initialized: now 1790297837607957 us; error 34 us; skew 500 ppm
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:17.614281 2900 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:57:17.615262 2900 webserver.cc:460] Webserver started at http://127.16.202.1:39849/ using document root <none> and password file <none>
I20260925 00:57:17.615836 2900 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:17.615907 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:17.616119 2900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:17.618307 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "cafb1bbbc1324506b44686cd912b10da"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "e9c82b7cf0d1c974ffe41158c16b1a44"
server_key_iv: "a1313dd4f7a4a3e2f2daa32cb4ae9ecd"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.618870 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "cafb1bbbc1324506b44686cd912b10da"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "e9c82b7cf0d1c974ffe41158c16b1a44"
server_key_iv: "a1313dd4f7a4a3e2f2daa32cb4ae9ecd"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.622711 2900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 00:57:17.625255 2915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.626557 2900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:17.626677 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "cafb1bbbc1324506b44686cd912b10da"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "e9c82b7cf0d1c974ffe41158c16b1a44"
server_key_iv: "a1313dd4f7a4a3e2f2daa32cb4ae9ecd"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.626780 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:17.649852 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:17.653529 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:17.653752 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:17.654418 2900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:17.655387 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:17.655436 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.655525 2900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:17.655548 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.665258 2900 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:44085
I20260925 00:57:17.665280 3028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:44085 every 8 connection(s)
I20260925 00:57:17.666251 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:57:17.672689 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 2900
I20260925 00:57:17.672830 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:57:17.673168 17192 external_mini_cluster.cc:1428] Setting key c3e20156dafbe35ed5ce3b72eb41306e
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:17.681075 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:17.668303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:44371 (local address 127.16.202.62:38515)
0925 00:57:17.668443 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:17.668446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:17.669329 (+ 883us) server_negotiation.cc:407] Connection header received
0925 00:57:17.670710 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:17.670713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:17.670771 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:17.670842 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:17.672069 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.672718 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.674135 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.674379 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.674474 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:17.677576 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:17.677580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:17.677582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:17.677610 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:17.678982 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:17.679524 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:17.679527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:17.679528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:17.679607 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:17.679937 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:17.679940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:17.679942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:17.680139 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:17.680227 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:17.680774 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 00:57:17.680909 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:57:17.681969 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:17.668656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38515 (local address 127.16.202.1:44371)
0925 00:57:17.669170 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:17.669217 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:17.670468 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:17.671035 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:17.671042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:17.671415 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:17.671913 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:17.671927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.672850 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.672854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:17.673999 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:17.674012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.674540 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.674544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:17.674755 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:17.675404 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:17.675426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:17.677315 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:17.679112 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:17.679119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:17.679139 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:17.679409 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:17.679693 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:17.679696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:17.679698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:17.679847 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:17.680241 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:17.680248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:17.680521 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 00:57:17.680729 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 00:57:17.681095 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:57:17.683431 3029 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38515
I20260925 00:57:17.683668 3029 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:17.684152 3029 heartbeater.cc:511] Master 127.16.202.62:38515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:57:17.685817 2838 ts_manager.cc:194] Registered new tserver with Master: cafb1bbbc1324506b44686cd912b10da (127.16.202.1:44085)
I20260925 00:57:17.687255 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:44371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:57:17.701064 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38515
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:17.809860 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:17.810138 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:17.810199 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:17.814291 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:17.814366 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:17.814455 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:57:17.819872 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:17.820987 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:17.822037 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:17.832085 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:17.832098 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:17.832300 3034 server_base.cc:1034] running on GCE node
W20260925 00:57:17.832108 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:17.832731 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:17.833403 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:17.834573 3034 hybrid_clock.cc:648] HybridClock initialized: now 1790297837834549 us; error 32 us; skew 500 ppm
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:17.837289 3034 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:57:17.838187 3034 webserver.cc:460] Webserver started at http://127.16.202.2:44697/ using document root <none> and password file <none>
I20260925 00:57:17.838745 3034 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:17.838815 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:17.839020 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:17.841133 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "a5ba4093c21d43aba47ba9e623f099fe"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "3f1dff61a56d7e26f6a49b6f5598a007"
server_key_iv: "4fed1a398dc00e2ad3826e6883c3e692"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.841712 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "a5ba4093c21d43aba47ba9e623f099fe"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "3f1dff61a56d7e26f6a49b6f5598a007"
server_key_iv: "4fed1a398dc00e2ad3826e6883c3e692"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.845881 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:17.848564 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.850016 3034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:17.850158 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "a5ba4093c21d43aba47ba9e623f099fe"
format_stamp: "Formatted at 2026-09-25 00:57:17 on dist-test-slave-rpk1"
server_key: "3f1dff61a56d7e26f6a49b6f5598a007"
server_key_iv: "4fed1a398dc00e2ad3826e6883c3e692"
server_key_version: "encryptionkey@0"
I20260925 00:57:17.850274 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:17.862850 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:17.865907 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:17.866093 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:17.866679 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:17.867542 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:17.867589 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.867641 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:17.867667 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:17.875928 3034 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:36157
I20260925 00:57:17.875963 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:36157 every 8 connection(s)
I20260925 00:57:17.876850 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:57:17.877372 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3034
I20260925 00:57:17.877483 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:57:17.877766 17192 external_mini_cluster.cc:1428] Setting key 1537d54b8f47540cdc8eb1457fb28a2d
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
Sep 25 00:57:17 dist-test-slave-rpk1 krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:57:17.891706 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:17.881110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:56267 (local address 127.16.202.62:38515)
0925 00:57:17.881257 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:17.881261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:17.881273 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:57:17.881309 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:17.881311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:17.881349 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:17.881419 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:17.882710 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.883302 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.884481 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.884803 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.884907 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:17.887675 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:17.887679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:17.887681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:17.887710 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:17.889285 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:17.889968 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:17.889971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:17.889973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:17.890078 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:17.890408 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:17.890453 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:17.890455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:17.890695 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:17.890791 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:17.891356 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 00:57:17.891540 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:57:17.892529 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:17.879128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38515 (local address 127.16.202.2:56267)
0925 00:57:17.879649 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:17.879692 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:17.880920 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:17.881634 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:17.881642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:17.882035 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:17.882556 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:17.882570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.883435 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.883438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:17.884344 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:17.884364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:17.884959 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:17.884963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:17.885163 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:17.885746 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:17.885763 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:17.887298 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:17.889436 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:17.889445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:17.889466 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:17.889849 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:17.890168 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:17.890170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:17.890172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:17.890328 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:17.890796 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:17.890802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:17.891093 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 00:57:17.891276 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 00:57:17.891638 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":155,"threads_started":1}
I20260925 00:57:17.893841 3163 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38515
I20260925 00:57:17.894203 3163 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:17.894936 3163 heartbeater.cc:511] Master 127.16.202.62:38515 requested a full tablet report, sending...
I20260925 00:57:17.896370 2838 ts_manager.cc:194] Registered new tserver with Master: a5ba4093c21d43aba47ba9e623f099fe (127.16.202.2:36157)
I20260925 00:57:17.897249 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:56267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:57:17.908108 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38515
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:18.019449 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:18.019753 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:18.019799 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:18.023950 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:18.024019 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:18.024080 3168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:57:18.029418 3168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:18.030467 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:18.031500 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:18.040946 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:18.041051 3168 server_base.cc:1034] running on GCE node
W20260925 00:57:18.040946 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:18.040946 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:18.041646 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:18.042265 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:18.043465 3168 hybrid_clock.cc:648] HybridClock initialized: now 1790297838043449 us; error 29 us; skew 500 ppm
Sep 25 00:57:18 dist-test-slave-rpk1 krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:18.046301 3168 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:57:18.047224 3168 webserver.cc:460] Webserver started at http://127.16.202.3:34505/ using document root <none> and password file <none>
I20260925 00:57:18.047783 3168 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:18.047849 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:18.048039 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:18.050144 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "ea50c99203ea47fc86709e9e62f57ef1"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "0f7914d993d7ef030e948c224f9f4887"
server_key_iv: "af9ccf3fe40f141b61fa608b82ac2d9d"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.050681 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "ea50c99203ea47fc86709e9e62f57ef1"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "0f7914d993d7ef030e948c224f9f4887"
server_key_iv: "af9ccf3fe40f141b61fa608b82ac2d9d"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.054569 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:18.057331 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.058662 3168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:18.058802 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "ea50c99203ea47fc86709e9e62f57ef1"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "0f7914d993d7ef030e948c224f9f4887"
server_key_iv: "af9ccf3fe40f141b61fa608b82ac2d9d"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.058923 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:18.073482 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:18.076849 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:18.077044 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:18.077642 3168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:18.078470 3168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:18.078516 3168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.078569 3168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:18.078595 3168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.087010 3168 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:45337
I20260925 00:57:18.087037 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:45337 every 8 connection(s)
I20260925 00:57:18.087910 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:57:18.094735 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3168
I20260925 00:57:18.094852 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:57:18.095101 17192 external_mini_cluster.cc:1428] Setting key 25533ef3b9fdc52924bea60865b562ad
Sep 25 00:57:18 dist-test-slave-rpk1 krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:18.101732 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.089707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:37285 (local address 127.16.202.62:38515)
0925 00:57:18.089847 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.089851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.090757 (+ 906us) server_negotiation.cc:407] Connection header received
0925 00:57:18.092064 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.092067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.092115 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.092188 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:18.093447 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.093977 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.095286 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.095512 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.095613 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.098438 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:18.098441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:18.098443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:18.098468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:18.099807 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:18.100288 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:18.100292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:18.100293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:18.100373 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:18.100698 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:18.100701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:18.100703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:18.100887 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:18.100971 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.101451 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.101574 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:57:18.102519 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.090133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38515 (local address 127.16.202.3:37285)
0925 00:57:18.090604 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.090647 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.091873 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.092335 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.092342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.092827 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:18.093299 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.093314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.094116 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.094119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.095167 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.095178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.095650 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.095654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.095852 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.096543 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:18.096568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:18.098272 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:18.099911 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:18.099917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:18.099934 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:18.100200 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:18.100456 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:18.100459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:18.100461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:18.100593 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:18.100982 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:18.100989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:18.101231 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.101398 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.101715 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260925 00:57:18.103628 3297 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38515
I20260925 00:57:18.103857 3297 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:18.104311 3297 heartbeater.cc:511] Master 127.16.202.62:38515 requested a full tablet report, sending...
I20260925 00:57:18.105331 2838 ts_manager.cc:194] Registered new tserver with Master: ea50c99203ea47fc86709e9e62f57ef1 (127.16.202.3:45337)
I20260925 00:57:18.105878 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:37285
I20260925 00:57:18.109267 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:57:18.117959 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.111192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.16.202.62:38515)
0925 00:57:18.111297 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.111300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.111428 (+ 128us) server_negotiation.cc:407] Connection header received
0925 00:57:18.111520 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.111522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.111561 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.111625 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:18.112290 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.112875 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.113643 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.113853 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.113928 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.115278 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:18.115280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:18.115283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:18.115303 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:18.116544 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:18.116979 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:18.116982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:18.116983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:18.117049 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:18.117276 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:18.117278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:18.117279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:18.117498 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:18.117610 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.117768 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.117837 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 00:57:18.120757 2838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:57:18.123721 2838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:57:18.138434 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.132951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:45337 (local address 127.0.0.1:47300)
0925 00:57:18.133769 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.133781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.133853 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.134513 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.134516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.134535 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:18.134624 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.134629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.135839 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.135843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.136755 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.136766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.137815 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.137819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.137936 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.137984 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.138084 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.138241 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":19,"thread_start_us":91,"threads_started":1}
I20260925 00:57:18.138463 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.132271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.16.202.1:44085)
0925 00:57:18.132626 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.132642 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.133101 (+ 459us) server_negotiation.cc:407] Connection header received
0925 00:57:18.133189 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.133195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.133422 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.133582 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:18.134471 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.135279 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.137307 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.137938 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.138042 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.138180 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.138223 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.138307 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":79,"threads_started":1}
I20260925 00:57:18.138463 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.133419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.16.202.3:45337)
0925 00:57:18.134081 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.134090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.134106 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:57:18.134177 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.134182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.134354 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.134546 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:18.134742 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.135728 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.136880 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.137686 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.137797 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.137949 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.138214 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.138308 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":93,"threads_started":1}
I20260925 00:57:18.138434 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.132169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44085 (local address 127.0.0.1:47404)
0925 00:57:18.132749 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.132799 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.132954 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.133843 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.133849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.133885 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:18.134261 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.134268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.135396 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.135399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.137186 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.137195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.138042 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.138046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.138119 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.138137 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.138186 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.138242 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":93,"spinlock_wait_cycles":4736,"thread_start_us":106,"threads_started":1}
I20260925 00:57:18.139216 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.132250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:36157 (local address 127.0.0.1:54350)
0925 00:57:18.133287 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.133300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.133400 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.133843 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.133849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.133887 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:18.134055 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.134063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.135506 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.135509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.136944 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.136954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.137947 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.137951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.138041 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.138066 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.138114 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.139083 (+ 969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":93,"spinlock_wait_cycles":7552,"thread_start_us":108,"threads_started":1}
I20260925 00:57:18.141462 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.132897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.16.202.2:36157)
0925 00:57:18.133264 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.133280 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.133381 (+ 101us) server_negotiation.cc:407] Connection header received
0925 00:57:18.133507 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.133512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.133696 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.133854 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:18.134204 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.135381 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.137068 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.137821 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.141030 (+ 3209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.141176 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.141227 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.141320 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
I20260925 00:57:18.141280 2963 tablet_service.cc:1467] Processing CreateTablet for tablet dbff4c4cb92241d38c75b3b4b0b26338 (DEFAULT_TABLE table=test-table [id=f2ee1a301e4b4d9c889ad23c35063aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:18.141314 3231 tablet_service.cc:1467] Processing CreateTablet for tablet dbff4c4cb92241d38c75b3b4b0b26338 (DEFAULT_TABLE table=test-table [id=f2ee1a301e4b4d9c889ad23c35063aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:18.142287 2963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbff4c4cb92241d38c75b3b4b0b26338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:18.142313 3231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbff4c4cb92241d38c75b3b4b0b26338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:18.144157 3097 tablet_service.cc:1467] Processing CreateTablet for tablet dbff4c4cb92241d38c75b3b4b0b26338 (DEFAULT_TABLE table=test-table [id=f2ee1a301e4b4d9c889ad23c35063aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:18.145293 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbff4c4cb92241d38c75b3b4b0b26338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:18.148775 3314 tablet_bootstrap.cc:492] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Bootstrap starting.
I20260925 00:57:18.150309 3315 tablet_bootstrap.cc:492] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: Bootstrap starting.
I20260925 00:57:18.152031 3314 tablet_bootstrap.cc:654] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:18.152758 3315 tablet_bootstrap.cc:654] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:18.153044 3316 tablet_bootstrap.cc:492] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Bootstrap starting.
I20260925 00:57:18.153347 3314 log.cc:824] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:18.154006 3315 log.cc:824] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:18.155601 3316 tablet_bootstrap.cc:654] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:18.155628 3314 tablet_bootstrap.cc:492] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: No bootstrap required, opened a new log
I20260925 00:57:18.155843 3314 ts_tablet_manager.cc:1397] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 00:57:18.156337 3315 tablet_bootstrap.cc:492] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: No bootstrap required, opened a new log
I20260925 00:57:18.156541 3315 ts_tablet_manager.cc:1397] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:57:18.157222 3316 log.cc:824] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:18.159513 3316 tablet_bootstrap.cc:492] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: No bootstrap required, opened a new log
I20260925 00:57:18.159724 3316 ts_tablet_manager.cc:1397] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:57:18.159819 3314 raft_consensus.cc:348] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.160087 3314 raft_consensus.cc:374] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:18.160140 3314 raft_consensus.cc:729] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea50c99203ea47fc86709e9e62f57ef1, State: Initialized, Role: FOLLOWER
I20260925 00:57:18.160341 3315 raft_consensus.cc:348] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.160616 3315 raft_consensus.cc:374] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:18.160560 3314 consensus_queue.cc:260] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.160712 3315 raft_consensus.cc:729] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafb1bbbc1324506b44686cd912b10da, State: Initialized, Role: FOLLOWER
I20260925 00:57:18.161164 3315 consensus_queue.cc:260] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.161227 3297 heartbeater.cc:503] Master 127.16.202.62:38515 was elected leader, sending a full tablet report...
I20260925 00:57:18.161644 3314 ts_tablet_manager.cc:1428] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:57:18.162115 3315 ts_tablet_manager.cc:1428] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:57:18.162340 3029 heartbeater.cc:503] Master 127.16.202.62:38515 was elected leader, sending a full tablet report...
I20260925 00:57:18.163398 3316 raft_consensus.cc:348] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.163681 3316 raft_consensus.cc:374] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:18.163743 3316 raft_consensus.cc:729] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ba4093c21d43aba47ba9e623f099fe, State: Initialized, Role: FOLLOWER
I20260925 00:57:18.164238 3316 consensus_queue.cc:260] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.164912 3163 heartbeater.cc:503] Master 127.16.202.62:38515 was elected leader, sending a full tablet report...
I20260925 00:57:18.165244 3316 ts_tablet_manager.cc:1428] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260925 00:57:18.167521 3030 tablet.cc:2369] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:18.218436 3321 raft_consensus.cc:482] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:18.218674 3321 raft_consensus.cc:504] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.219799 3321 leader_election.cc:288] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ba4093c21d43aba47ba9e623f099fe (127.16.202.2:36157), ea50c99203ea47fc86709e9e62f57ef1 (127.16.202.3:45337)
I20260925 00:57:18.223716 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.220210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:50037 (local address 127.16.202.3:45337)
0925 00:57:18.220375 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.220380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.220544 (+ 164us) server_negotiation.cc:407] Connection header received
0925 00:57:18.220706 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.220710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.220751 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.220811 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:18.221116 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.221598 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.222764 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.223167 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.223274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.223327 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.223507 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.223577 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 00:57:18.223737 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.220153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:54347 (local address 127.16.202.2:36157)
0925 00:57:18.220277 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.220282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.220496 (+ 214us) server_negotiation.cc:407] Connection header received
0925 00:57:18.220632 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.220635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.220704 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.220775 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:18.221103 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.221600 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.222765 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.223175 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.223283 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.223341 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.223504 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.223619 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:18.223757 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.220089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:45337 (local address 127.16.202.1:50037)
0925 00:57:18.220468 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.220481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.220588 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.220831 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.220834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.220873 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:18.221009 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.221014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.221696 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.221699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.222643 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.222652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.223274 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.223278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.223384 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.223404 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.223471 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.223591 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":97,"spinlock_wait_cycles":2688,"thread_start_us":123,"threads_started":1}
I20260925 00:57:18.223763 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.220059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:36157 (local address 127.16.202.1:54347)
0925 00:57:18.220392 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.220408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.220514 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.220794 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.220797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.220821 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:18.220989 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.220996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.221704 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.221707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.222639 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.222648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.223287 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.223290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.223397 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.223415 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.223478 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.223600 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":80,"spinlock_wait_cycles":15616,"thread_start_us":93,"threads_started":1}
I20260925 00:57:18.224462 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbff4c4cb92241d38c75b3b4b0b26338" candidate_uuid: "cafb1bbbc1324506b44686cd912b10da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ba4093c21d43aba47ba9e623f099fe" is_pre_election: true
I20260925 00:57:18.224462 3251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbff4c4cb92241d38c75b3b4b0b26338" candidate_uuid: "cafb1bbbc1324506b44686cd912b10da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea50c99203ea47fc86709e9e62f57ef1" is_pre_election: true
I20260925 00:57:18.224763 3251 raft_consensus.cc:2454] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cafb1bbbc1324506b44686cd912b10da in term 0.
I20260925 00:57:18.224763 3117 raft_consensus.cc:2454] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cafb1bbbc1324506b44686cd912b10da in term 0.
I20260925 00:57:18.225225 2918 leader_election.cc:302] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ba4093c21d43aba47ba9e623f099fe, cafb1bbbc1324506b44686cd912b10da; no voters:
I20260925 00:57:18.225486 3321 raft_consensus.cc:2781] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:57:18.225574 3321 raft_consensus.cc:482] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:57:18.225621 3321 raft_consensus.cc:3037] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:18.226630 3321 raft_consensus.cc:504] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.226960 3321 leader_election.cc:288] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [CANDIDATE]: Term 1 election: Requested vote from peers a5ba4093c21d43aba47ba9e623f099fe (127.16.202.2:36157), ea50c99203ea47fc86709e9e62f57ef1 (127.16.202.3:45337)
I20260925 00:57:18.227325 3251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbff4c4cb92241d38c75b3b4b0b26338" candidate_uuid: "cafb1bbbc1324506b44686cd912b10da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea50c99203ea47fc86709e9e62f57ef1"
I20260925 00:57:18.227325 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbff4c4cb92241d38c75b3b4b0b26338" candidate_uuid: "cafb1bbbc1324506b44686cd912b10da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ba4093c21d43aba47ba9e623f099fe"
I20260925 00:57:18.227435 3251 raft_consensus.cc:3037] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:18.227484 3117 raft_consensus.cc:3037] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:18.228506 3251 raft_consensus.cc:2454] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cafb1bbbc1324506b44686cd912b10da in term 1.
I20260925 00:57:18.228590 3117 raft_consensus.cc:2454] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cafb1bbbc1324506b44686cd912b10da in term 1.
I20260925 00:57:18.228876 2917 leader_election.cc:302] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cafb1bbbc1324506b44686cd912b10da, ea50c99203ea47fc86709e9e62f57ef1; no voters:
I20260925 00:57:18.229066 3321 raft_consensus.cc:2781] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:18.229290 3321 raft_consensus.cc:686] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [term 1 LEADER]: Becoming Leader. State: Replica: cafb1bbbc1324506b44686cd912b10da, State: Running, Role: LEADER
I20260925 00:57:18.229621 3321 consensus_queue.cc:237] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } }
I20260925 00:57:18.233196 2838 catalog_manager.cc:5495] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da reported cstate change: term changed from 0 to 1, leader changed from <none> to cafb1bbbc1324506b44686cd912b10da (127.16.202.1). New cstate: current_term: 1 leader_uuid: "cafb1bbbc1324506b44686cd912b10da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafb1bbbc1324506b44686cd912b10da" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 } health_report { overall_health: UNKNOWN } } }
I20260925 00:57:18.265722 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.262042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.16.202.1:44085)
0925 00:57:18.262189 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.262194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.262225 (+ 31us) server_negotiation.cc:407] Connection header received
0925 00:57:18.262300 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.262302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.262371 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.262503 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:18.262748 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.263297 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.264193 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.264430 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.264524 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.264871 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:18.265302 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:18.265437 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.265539 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.265579 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:57:18.271931 3117 raft_consensus.cc:1261] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 1 FOLLOWER]: Refusing update from remote peer cafb1bbbc1324506b44686cd912b10da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:18.271931 3251 raft_consensus.cc:1261] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 1 FOLLOWER]: Refusing update from remote peer cafb1bbbc1324506b44686cd912b10da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:18.272647 3321 consensus_queue.cc:1035] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ba4093c21d43aba47ba9e623f099fe" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:18.272882 3325 consensus_queue.cc:1035] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea50c99203ea47fc86709e9e62f57ef1" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:18.280781 3329 mvcc.cc:205] Tried to move back new op lower bound from 7333059945554690048 to 7333059945391386624. Current Snapshot: MvccSnapshot[applied={T|T < 7333059945554690048}]
I20260925 00:57:18.282119 3330 mvcc.cc:205] Tried to move back new op lower bound from 7333059945554690048 to 7333059945391386624. Current Snapshot: MvccSnapshot[applied={T|T < 7333059945554690048}]
I20260925 00:57:18.289151 2836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:18.289368 2836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:18.292136 2836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f0ba794dfee464688b4085c87eec089: Sending DeleteTablet for 3 replicas of tablet dbff4c4cb92241d38c75b3b4b0b26338
I20260925 00:57:18.293216 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet dbff4c4cb92241d38c75b3b4b0b26338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:54350
I20260925 00:57:18.293260 2963 tablet_service.cc:1514] Processing DeleteTablet for tablet dbff4c4cb92241d38c75b3b4b0b26338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:47404
I20260925 00:57:18.293845 3338 tablet_replica.cc:330] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: stopping tablet replica
I20260925 00:57:18.293876 3231 tablet_service.cc:1514] Processing DeleteTablet for tablet dbff4c4cb92241d38c75b3b4b0b26338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:18 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260925 00:57:18.294338 3340 tablet_replica.cc:330] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: stopping tablet replica
I20260925 00:57:18.294636 3339 tablet_replica.cc:330] T dbff4c4cb92241d38c75b3b4b0b26338 P cafb1bbbc1324506b44686cd912b10da: stopping tablet replica
I20260925 00:57:18.294651 3340 raft_consensus.cc:2229] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:18.294708 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2900
I20260925 00:57:18.295920 3340 raft_consensus.cc:2258] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:18.294354 3338 raft_consensus.cc:2229] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:18.297102 3338 raft_consensus.cc:2258] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:18.297632 3340 ts_tablet_manager.cc:1905] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:18.298511 3338 ts_tablet_manager.cc:1905] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:18.300311 3340 ts_tablet_manager.cc:1918] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:18.300420 3340 log.cc:1196] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/dbff4c4cb92241d38c75b3b4b0b26338
I20260925 00:57:18.300756 3340 ts_tablet_manager.cc:1939] T dbff4c4cb92241d38c75b3b4b0b26338 P ea50c99203ea47fc86709e9e62f57ef1: Deleting consensus metadata
I20260925 00:57:18.301024 3338 ts_tablet_manager.cc:1918] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:18.301118 3338 log.cc:1196] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/dbff4c4cb92241d38c75b3b4b0b26338
I20260925 00:57:18.301460 3338 ts_tablet_manager.cc:1939] T dbff4c4cb92241d38c75b3b4b0b26338 P a5ba4093c21d43aba47ba9e623f099fe: Deleting consensus metadata
I20260925 00:57:18.302210 2823 catalog_manager.cc:4841] TS ea50c99203ea47fc86709e9e62f57ef1 (127.16.202.3:45337): tablet dbff4c4cb92241d38c75b3b4b0b26338 (table test-table [id=f2ee1a301e4b4d9c889ad23c35063aa8]) successfully deleted
I20260925 00:57:18.302608 2824 catalog_manager.cc:4841] TS a5ba4093c21d43aba47ba9e623f099fe (127.16.202.2:36157): tablet dbff4c4cb92241d38c75b3b4b0b26338 (table test-table [id=f2ee1a301e4b4d9c889ad23c35063aa8]) successfully deleted
W20260925 00:57:18.302755 2823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:44085 (error 108)
I20260925 00:57:18.303004 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3034
I20260925 00:57:18.304121 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.303800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44085 (local address 127.0.0.1:47422)
0925 00:57:18.303970 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.304035 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 00:57:18.304270 2823 catalog_manager.cc:4584] TS cafb1bbbc1324506b44686cd912b10da (127.16.202.1:44085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbff4c4cb92241d38c75b3b4b0b26338: Network error: Client connection negotiation failed: client connection to 127.16.202.1:44085: connect: Connection refused (error 111)
I20260925 00:57:18.308077 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3168
I20260925 00:57:18.312984 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 2806
2026-09-25T00:57:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2262 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T00:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:18Z Disabled control of system clock
I20260925 00:57:18.342056 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36095
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:57:18.454605 3345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:18.454936 3345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:18.454972 3345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:18.459337 3345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:18.459421 3345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:18.459436 3345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:18.459452 3345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:18.459467 3345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:18.464629 3345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:36095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:18.465798 3345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:18.467275 3345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:18.476943 3351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:18.477039 3345 server_base.cc:1034] running on GCE node
W20260925 00:57:18.476882 3350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:18.476955 3353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:18.477890 3345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:18.479068 3345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:18.480248 3345 hybrid_clock.cc:648] HybridClock initialized: now 1790297838480223 us; error 41 us; skew 500 ppm
I20260925 00:57:18.482249 3345 webserver.cc:460] Webserver started at http://127.16.202.62:44273/ using document root <none> and password file <none>
I20260925 00:57:18.482812 3345 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:18.482887 3345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:18.483068 3345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:18.485353 3345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "7e547994e8724941935e7f8f3f15af68"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "e5679258d783bfc040d32efde59a1810"
server_key_iv: "816346619e5ea198d8937243ff05c7b7"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.485934 3345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "7e547994e8724941935e7f8f3f15af68"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "e5679258d783bfc040d32efde59a1810"
server_key_iv: "816346619e5ea198d8937243ff05c7b7"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.489913 3345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:18.492571 3359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.494139 3345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:18.494285 3345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "7e547994e8724941935e7f8f3f15af68"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "e5679258d783bfc040d32efde59a1810"
server_key_iv: "816346619e5ea198d8937243ff05c7b7"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.494436 3345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:18.504626 3345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:18.505445 3345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:18.505623 3345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:18.512784 3345 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:36095
I20260925 00:57:18.512809 3411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:36095 every 8 connection(s)
I20260925 00:57:18.513836 3345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:18.517077 3412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:18.518146 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3345
I20260925 00:57:18.518249 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:18.518540 17192 external_mini_cluster.cc:1428] Setting key cf4db872fda995ea6af904d7cfb0323a
I20260925 00:57:18.523211 3412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68: Bootstrap starting.
I20260925 00:57:18.525933 3412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:18.526690 3412 log.cc:824] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:18.527962 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.519905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.16.202.62:36095)
0925 00:57:18.520530 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.520545 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.520581 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:57:18.521275 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.521294 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.521711 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.522000 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:18.522352 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.523274 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.524483 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.525010 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.525157 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.526071 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:18.526079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:18.526098 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:18.526551 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:18.526613 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:18.526635 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:18.526716 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.526928 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.527211 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260925 00:57:18.528752 3412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68: No bootstrap required, opened a new log
I20260925 00:57:18.532485 3412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e547994e8724941935e7f8f3f15af68" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36095 } }
I20260925 00:57:18.532755 3412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:18.532823 3412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e547994e8724941935e7f8f3f15af68, State: Initialized, Role: FOLLOWER
I20260925 00:57:18.533245 3412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e547994e8724941935e7f8f3f15af68" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36095 } }
I20260925 00:57:18.533398 3412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:18.533463 3412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:18.533558 3412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:18.534510 3412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e547994e8724941935e7f8f3f15af68" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36095 } }
I20260925 00:57:18.534843 3412 leader_election.cc:302] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e547994e8724941935e7f8f3f15af68; no voters:
I20260925 00:57:18.535136 3412 leader_election.cc:288] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:18.535318 3417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:18.535642 3417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [term 1 LEADER]: Becoming Leader. State: Replica: 7e547994e8724941935e7f8f3f15af68, State: Running, Role: LEADER
I20260925 00:57:18.536019 3417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e547994e8724941935e7f8f3f15af68" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36095 } }
I20260925 00:57:18.536439 3412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:18.537622 3418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e547994e8724941935e7f8f3f15af68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e547994e8724941935e7f8f3f15af68" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36095 } } }
I20260925 00:57:18.537755 3418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:18.538079 3419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e547994e8724941935e7f8f3f15af68. Latest consensus state: current_term: 1 leader_uuid: "7e547994e8724941935e7f8f3f15af68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e547994e8724941935e7f8f3f15af68" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 36095 } } }
I20260925 00:57:18.538187 3419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:18.538134 3426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:18.541019 3426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:18.546540 3426 catalog_manager.cc:1347] Generated new cluster ID: 3cbba0cab50d4bb692166c9965cd051c
I20260925 00:57:18.546617 3426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:18.568791 3426 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:18.569944 3426 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:18.583938 3426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68: Generated new TSK 0
I20260925 00:57:18.584734 3426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:18.595220 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36095
--builtin_ntp_servers=127.16.202.20:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:57:18.711843 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:18.712121 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:18.712162 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:18.716322 3436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:18.716391 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:18.716449 3436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:57:18.721395 3436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:18.722503 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:18.723889 3436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:18.733341 3441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:18.733439 3442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:18.733439 3444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:18.733768 3436 server_base.cc:1034] running on GCE node
I20260925 00:57:18.734145 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:18.734776 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:18.735935 3436 hybrid_clock.cc:648] HybridClock initialized: now 1790297838735915 us; error 37 us; skew 500 ppm
I20260925 00:57:18.737916 3436 webserver.cc:460] Webserver started at http://127.16.202.1:33705/ using document root <none> and password file <none>
I20260925 00:57:18.738551 3436 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:18.738626 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:18.738824 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:18.740938 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "770df6ef0ded4aceb3a70e5b0f614efd"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "041ef221cc7f33e986424fee77d411c8"
server_key_iv: "cc703228b4c2a5a2600a03816ec2fb77"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.741494 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "770df6ef0ded4aceb3a70e5b0f614efd"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "041ef221cc7f33e986424fee77d411c8"
server_key_iv: "cc703228b4c2a5a2600a03816ec2fb77"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.745299 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:18.747845 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.749152 3436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:18.749270 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "770df6ef0ded4aceb3a70e5b0f614efd"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "041ef221cc7f33e986424fee77d411c8"
server_key_iv: "cc703228b4c2a5a2600a03816ec2fb77"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.749368 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:18.782830 3436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:18.783605 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:18.783797 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:18.784415 3436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:18.785506 3436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:18.785558 3436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.785614 3436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:18.785641 3436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.794436 3436 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:41979
I20260925 00:57:18.794487 3563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:41979 every 8 connection(s)
I20260925 00:57:18.795364 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:57:18.802294 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3436
I20260925 00:57:18.802419 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:57:18.802676 17192 external_mini_cluster.cc:1428] Setting key 2e34d80be65519c3ac6865c45dfe3be2
I20260925 00:57:18.804617 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36095
--builtin_ntp_servers=127.16.202.20:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:57:18.805214 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.797335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:50039 (local address 127.16.202.62:36095)
0925 00:57:18.797491 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.797495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.798657 (+ 1162us) server_negotiation.cc:407] Connection header received
0925 00:57:18.800109 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.800112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.800176 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.800252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:18.801257 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.801798 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.803015 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.803310 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.803408 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.804560 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:18.804564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:18.804566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:18.804580 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:18.804605 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:18.804615 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:18.804729 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.804989 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.805071 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:57:18.806129 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.797739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36095 (local address 127.16.202.1:50039)
0925 00:57:18.798506 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.798551 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.799892 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.800443 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.800451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.800542 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:18.801114 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.801130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.801924 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.801927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.802900 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.802911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.803477 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.803481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.803639 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.804284 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:18.804286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:18.804330 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:18.804400 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:18.804839 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:18.804849 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:18.804855 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.804967 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.805233 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":121,"threads_started":1}
I20260925 00:57:18.807395 3564 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36095
I20260925 00:57:18.807664 3564 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:18.808197 3564 heartbeater.cc:511] Master 127.16.202.62:36095 requested a full tablet report, sending...
I20260925 00:57:18.809744 3376 ts_manager.cc:194] Registered new tserver with Master: 770df6ef0ded4aceb3a70e5b0f614efd (127.16.202.1:41979)
I20260925 00:57:18.810925 3376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.1:50039
W20260925 00:57:18.914896 3567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:18.915170 3567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:18.915210 3567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:18.919257 3567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:18.919315 3567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:18.919371 3567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:57:18.924163 3567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:18.925360 3567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:18.926777 3567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:57:18.936569 3567 server_base.cc:1034] running on GCE node
W20260925 00:57:18.936499 3575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:18.936496 3573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:18.936758 3572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:18.937177 3567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:18.937844 3567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:18.939015 3567 hybrid_clock.cc:648] HybridClock initialized: now 1790297838938993 us; error 35 us; skew 500 ppm
I20260925 00:57:18.940806 3567 webserver.cc:460] Webserver started at http://127.16.202.2:33201/ using document root <none> and password file <none>
I20260925 00:57:18.941355 3567 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:18.941421 3567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:18.941630 3567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:18.943727 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "002e144cbd304ab393d3f940d12af213"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "b36ae7061756adab6844f59a4345612c"
server_key_iv: "85297555d7c397716c9c71d31de076b7"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.944290 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "002e144cbd304ab393d3f940d12af213"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "b36ae7061756adab6844f59a4345612c"
server_key_iv: "85297555d7c397716c9c71d31de076b7"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.948138 3567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 00:57:18.950861 3581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.952345 3567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:57:18.952486 3567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "002e144cbd304ab393d3f940d12af213"
format_stamp: "Formatted at 2026-09-25 00:57:18 on dist-test-slave-rpk1"
server_key: "b36ae7061756adab6844f59a4345612c"
server_key_iv: "85297555d7c397716c9c71d31de076b7"
server_key_version: "encryptionkey@0"
I20260925 00:57:18.952608 3567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:18.968472 3567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:18.969311 3567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:18.969514 3567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:18.970129 3567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:18.971172 3567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:18.971221 3567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.971280 3567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:18.971304 3567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:18.980139 3567 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:44251
I20260925 00:57:18.980175 3694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:44251 every 8 connection(s)
I20260925 00:57:18.981150 3567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:57:18.990434 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.983184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:53461 (local address 127.16.202.62:36095)
0925 00:57:18.983316 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:18.983320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:18.984399 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 00:57:18.985841 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:18.985844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:18.985907 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:18.985986 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:18.986891 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.987417 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.988542 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.988788 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.988879 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.989961 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:18.989965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:18.989967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:18.989982 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:18.990001 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:18.990009 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:18.990068 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:18.990213 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 00:57:18.990278 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:57:18.991291 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:18.983600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36095 (local address 127.16.202.2:53461)
0925 00:57:18.984247 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:18.984288 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:18.985626 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:18.986176 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:18.986183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:18.986270 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:18.986747 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.986760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.987547 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.987550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.988415 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:18.988425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:18.988930 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:18.988934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:18.989098 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:18.989689 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:18.989690 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:18.989731 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:18.989797 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:18.990078 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:18.990087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:18.990093 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:18.990201 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 00:57:18.990488 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260925 00:57:18.991439 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3567
I20260925 00:57:18.991569 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:57:18.991889 17192 external_mini_cluster.cc:1428] Setting key 9940cd2c3d7c8781426edfb0696f4b06
I20260925 00:57:18.992533 3695 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36095
I20260925 00:57:18.992817 3695 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:18.993384 3695 heartbeater.cc:511] Master 127.16.202.62:36095 requested a full tablet report, sending...
I20260925 00:57:18.993947 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36095
--builtin_ntp_servers=127.16.202.20:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:57:18.994426 3376 ts_manager.cc:194] Registered new tserver with Master: 002e144cbd304ab393d3f940d12af213 (127.16.202.2:44251)
I20260925 00:57:18.994976 3376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.2:53461
W20260925 00:57:19.105561 3698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:19.105842 3698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:19.105881 3698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:19.110023 3698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:19.110088 3698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:19.110145 3698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:57:19.115033 3698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:19.116123 3698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:19.117520 3698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:19.127060 3704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:19.127051 3698 server_base.cc:1034] running on GCE node
W20260925 00:57:19.127058 3703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:19.127084 3706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:19.127842 3698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:19.128508 3698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:19.129680 3698 hybrid_clock.cc:648] HybridClock initialized: now 1790297839129669 us; error 52 us; skew 500 ppm
I20260925 00:57:19.131662 3698 webserver.cc:460] Webserver started at http://127.16.202.3:45407/ using document root <none> and password file <none>
I20260925 00:57:19.132241 3698 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:19.132313 3698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:19.132539 3698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:19.134794 3698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "5f86a1390a0243a38d283a6fee99b49d"
format_stamp: "Formatted at 2026-09-25 00:57:19 on dist-test-slave-rpk1"
server_key: "a570fa6f55303617fa49cbac64b8a2a3"
server_key_iv: "5d38b1022255960869f8c4acfc4b5fa8"
server_key_version: "encryptionkey@0"
I20260925 00:57:19.135321 3698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "5f86a1390a0243a38d283a6fee99b49d"
format_stamp: "Formatted at 2026-09-25 00:57:19 on dist-test-slave-rpk1"
server_key: "a570fa6f55303617fa49cbac64b8a2a3"
server_key_iv: "5d38b1022255960869f8c4acfc4b5fa8"
server_key_version: "encryptionkey@0"
I20260925 00:57:19.139232 3698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:19.141660 3712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:19.142956 3698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:19.143075 3698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "5f86a1390a0243a38d283a6fee99b49d"
format_stamp: "Formatted at 2026-09-25 00:57:19 on dist-test-slave-rpk1"
server_key: "a570fa6f55303617fa49cbac64b8a2a3"
server_key_iv: "5d38b1022255960869f8c4acfc4b5fa8"
server_key_version: "encryptionkey@0"
I20260925 00:57:19.143187 3698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:19.157480 3698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:19.158223 3698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:19.158398 3698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:19.158999 3698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:19.159972 3698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:19.160020 3698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:19.160065 3698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:19.160094 3698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:19.168650 3698 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:43457
I20260925 00:57:19.168707 3825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:43457 every 8 connection(s)
I20260925 00:57:19.169620 3698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:57:19.170142 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3698
I20260925 00:57:19.170259 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:57:19.170545 17192 external_mini_cluster.cc:1428] Setting key 8f5ad0457f1a1c3dd063e1864e928889
I20260925 00:57:19.178983 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.171577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:42745 (local address 127.16.202.62:36095)
0925 00:57:19.171729 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.171734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.173028 (+ 1294us) server_negotiation.cc:407] Connection header received
0925 00:57:19.174432 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.174435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.174492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.174569 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:19.175486 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.176018 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.177206 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.177436 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.177534 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.178538 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:19.178542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:19.178544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:19.178557 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:19.178576 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:19.178583 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:19.178640 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.178775 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.178835 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:57:19.179840 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.172057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:36095 (local address 127.16.202.3:42745)
0925 00:57:19.172872 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.172938 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.174231 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.174745 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.174753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.174839 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:19.175351 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.175365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.176136 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.176140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.177095 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.177105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.177577 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.177581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.177739 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.178285 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:19.178286 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:19.178326 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:19.178389 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:19.178650 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:19.178659 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:19.178665 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.178769 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.179093 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":132,"threads_started":1}
I20260925 00:57:19.180969 3826 heartbeater.cc:347] Connected to a master server at 127.16.202.62:36095
I20260925 00:57:19.181177 3826 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:19.181659 3826 heartbeater.cc:511] Master 127.16.202.62:36095 requested a full tablet report, sending...
I20260925 00:57:19.182627 3376 ts_manager.cc:194] Registered new tserver with Master: 5f86a1390a0243a38d283a6fee99b49d (127.16.202.3:43457)
I20260925 00:57:19.183187 3376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.3:42745
I20260925 00:57:19.184544 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:57:19.190434 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.186581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.16.202.62:36095)
0925 00:57:19.186674 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.186678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.186851 (+ 173us) server_negotiation.cc:407] Connection header received
0925 00:57:19.186966 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.186968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.187002 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.187066 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:19.187265 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.187755 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.188602 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.188880 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.188967 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.189821 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:19.189825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:19.189828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:19.189843 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:19.189865 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:19.189876 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:19.190031 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.190248 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.190290 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 00:57:19.193352 3376 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:57:19.196481 3376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:57:19.210703 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.205980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.16.202.3:43457)
0925 00:57:19.206511 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.206523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.206547 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:57:19.206621 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.206625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.206796 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.206947 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.207115 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.208179 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.209363 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.210032 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.210129 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.210330 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.210428 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.210526 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":141,"threads_started":1}
I20260925 00:57:19.210703 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.205588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43457 (local address 127.0.0.1:55546)
0925 00:57:19.205855 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.205867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.205944 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.206935 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.206938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.206956 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.207025 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.207030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.208274 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.208277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.209234 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.209243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.210148 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.210152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.210250 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.210286 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.210400 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.210544 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":59,"thread_start_us":81,"threads_started":1}
I20260925 00:57:19.211040 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.204997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.16.202.1:41979)
0925 00:57:19.205861 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.205875 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.205899 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:57:19.205979 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.205984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.206202 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.206739 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.206873 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.207711 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.209341 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.210115 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.210217 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.210392 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.210767 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.210885 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":137,"threads_started":1}
I20260925 00:57:19.211297 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.204850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:41979 (local address 127.0.0.1:53704)
0925 00:57:19.205393 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.205444 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.205579 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.206380 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.206386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.206410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.206599 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.206607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.207811 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.207814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.209209 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.209219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.210449 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.210453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.210528 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.210547 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.210609 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.210667 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":41,"thread_start_us":77,"threads_started":1}
I20260925 00:57:19.211483 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.205507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.16.202.2:44251)
0925 00:57:19.205925 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.205936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.205981 (+ 45us) server_negotiation.cc:407] Connection header received
0925 00:57:19.206058 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.206063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.206254 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.206736 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.206874 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.208050 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.209409 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.210115 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.210914 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.211163 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.211218 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.211316 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":122,"threads_started":1}
I20260925 00:57:19.210716 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.204993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:44251 (local address 127.0.0.1:58880)
0925 00:57:19.205572 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.205585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.205674 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.206408 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.206411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.206438 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.206594 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.206601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.208169 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.208172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.209282 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.209291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.210250 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.210253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.210339 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.210373 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.210424 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.210538 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":22,"spinlock_wait_cycles":2816,"thread_start_us":121,"threads_started":1}
I20260925 00:57:19.213941 3760 tablet_service.cc:1467] Processing CreateTablet for tablet 57dbe03d30d14f708592ac42601472bf (DEFAULT_TABLE table=test-table [id=6640faf9d000475cb496aada97147097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:19.214113 3498 tablet_service.cc:1467] Processing CreateTablet for tablet 57dbe03d30d14f708592ac42601472bf (DEFAULT_TABLE table=test-table [id=6640faf9d000475cb496aada97147097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:19.214917 3760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57dbe03d30d14f708592ac42601472bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:19.215133 3498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57dbe03d30d14f708592ac42601472bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:19.215194 3629 tablet_service.cc:1467] Processing CreateTablet for tablet 57dbe03d30d14f708592ac42601472bf (DEFAULT_TABLE table=test-table [id=6640faf9d000475cb496aada97147097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:19.216130 3629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57dbe03d30d14f708592ac42601472bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:19.221017 3843 tablet_bootstrap.cc:492] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Bootstrap starting.
I20260925 00:57:19.222725 3844 tablet_bootstrap.cc:492] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: Bootstrap starting.
I20260925 00:57:19.223170 3845 tablet_bootstrap.cc:492] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Bootstrap starting.
I20260925 00:57:19.224596 3844 tablet_bootstrap.cc:654] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:19.224588 3843 tablet_bootstrap.cc:654] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:19.225297 3845 tablet_bootstrap.cc:654] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:19.225364 3844 log.cc:824] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:19.225441 3843 log.cc:824] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:19.226016 3845 log.cc:824] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:19.227368 3844 tablet_bootstrap.cc:492] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: No bootstrap required, opened a new log
I20260925 00:57:19.227568 3844 ts_tablet_manager.cc:1397] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:57:19.227919 3845 tablet_bootstrap.cc:492] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: No bootstrap required, opened a new log
I20260925 00:57:19.228106 3845 ts_tablet_manager.cc:1397] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:57:19.228482 3843 tablet_bootstrap.cc:492] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: No bootstrap required, opened a new log
I20260925 00:57:19.228652 3843 ts_tablet_manager.cc:1397] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 00:57:19.230964 3844 raft_consensus.cc:348] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.231185 3844 raft_consensus.cc:374] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:19.231237 3844 raft_consensus.cc:729] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770df6ef0ded4aceb3a70e5b0f614efd, State: Initialized, Role: FOLLOWER
I20260925 00:57:19.231652 3844 consensus_queue.cc:260] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.232517 3844 ts_tablet_manager.cc:1428] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:57:19.232372 3845 raft_consensus.cc:348] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.232498 3843 raft_consensus.cc:348] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.232653 3845 raft_consensus.cc:374] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:19.232721 3843 raft_consensus.cc:374] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:19.232725 3845 raft_consensus.cc:729] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002e144cbd304ab393d3f940d12af213, State: Initialized, Role: FOLLOWER
I20260925 00:57:19.232769 3843 raft_consensus.cc:729] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f86a1390a0243a38d283a6fee99b49d, State: Initialized, Role: FOLLOWER
I20260925 00:57:19.232887 3564 heartbeater.cc:503] Master 127.16.202.62:36095 was elected leader, sending a full tablet report...
I20260925 00:57:19.233161 3845 consensus_queue.cc:260] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.233161 3843 consensus_queue.cc:260] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.233793 3826 heartbeater.cc:503] Master 127.16.202.62:36095 was elected leader, sending a full tablet report...
I20260925 00:57:19.234107 3843 ts_tablet_manager.cc:1428] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:57:19.234304 3845 ts_tablet_manager.cc:1428] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:57:19.234499 3695 heartbeater.cc:503] Master 127.16.202.62:36095 was elected leader, sending a full tablet report...
W20260925 00:57:19.296574 3565 tablet.cc:2369] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:19.408349 3849 raft_consensus.cc:482] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:19.408581 3849 raft_consensus.cc:504] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.409832 3849 leader_election.cc:288] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 002e144cbd304ab393d3f940d12af213 (127.16.202.2:44251), 5f86a1390a0243a38d283a6fee99b49d (127.16.202.3:43457)
I20260925 00:57:19.412891 3851 raft_consensus.cc:482] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:19.413077 3851 raft_consensus.cc:504] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.414278 3851 leader_election.cc:288] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770df6ef0ded4aceb3a70e5b0f614efd (127.16.202.1:41979), 5f86a1390a0243a38d283a6fee99b49d (127.16.202.3:43457)
I20260925 00:57:19.414577 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.410216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:57417 (local address 127.16.202.3:43457)
0925 00:57:19.410336 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.410340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.410532 (+ 192us) server_negotiation.cc:407] Connection header received
0925 00:57:19.410609 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.410611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.410663 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.410742 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.411101 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.411873 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.413629 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.414057 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.414173 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.414224 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.414382 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.414455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:57:19.414600 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.410125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43457 (local address 127.16.202.1:57417)
0925 00:57:19.410453 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.410465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.410536 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.410761 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.410765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.410801 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.410994 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.411002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.412002 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.412006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.413475 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.413482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.414173 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.414177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.414273 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.414294 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.414356 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.414445 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":180,"spinlock_wait_cycles":27648,"thread_start_us":91,"threads_started":1}
I20260925 00:57:19.415107 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.410164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:39521 (local address 127.16.202.2:44251)
0925 00:57:19.410288 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.410292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.410501 (+ 209us) server_negotiation.cc:407] Connection header received
0925 00:57:19.410608 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.410611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.410663 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.410742 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.411101 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.411873 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.413576 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.414130 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.414243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.414289 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.414898 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.415005 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:57:19.415177 3780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57dbe03d30d14f708592ac42601472bf" candidate_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f86a1390a0243a38d283a6fee99b49d" is_pre_election: true
I20260925 00:57:19.415470 3780 raft_consensus.cc:2454] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770df6ef0ded4aceb3a70e5b0f614efd in term 0.
I20260925 00:57:19.415771 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.410069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:44251 (local address 127.16.202.1:39521)
0925 00:57:19.410372 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.410390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.410508 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.410761 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.410765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.410802 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.410999 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.411005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.412002 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.412006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.413439 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.413452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.414414 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.414420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.414522 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.414547 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.415394 (+ 847us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.415471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":100,"spinlock_wait_cycles":3840,"thread_start_us":69,"threads_started":1}
I20260925 00:57:19.416023 3453 leader_election.cc:302] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f86a1390a0243a38d283a6fee99b49d, 770df6ef0ded4aceb3a70e5b0f614efd; no voters:
I20260925 00:57:19.416347 3849 raft_consensus.cc:2781] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:57:19.416368 3649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57dbe03d30d14f708592ac42601472bf" candidate_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002e144cbd304ab393d3f940d12af213" is_pre_election: true
I20260925 00:57:19.416445 3849 raft_consensus.cc:482] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:57:19.416525 3849 raft_consensus.cc:3037] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:19.416587 3649 raft_consensus.cc:2454] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770df6ef0ded4aceb3a70e5b0f614efd in term 0.
I20260925 00:57:19.417804 3849 raft_consensus.cc:504] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.418143 3849 leader_election.cc:288] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [CANDIDATE]: Term 1 election: Requested vote from peers 002e144cbd304ab393d3f940d12af213 (127.16.202.2:44251), 5f86a1390a0243a38d283a6fee99b49d (127.16.202.3:43457)
I20260925 00:57:19.418431 3649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57dbe03d30d14f708592ac42601472bf" candidate_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002e144cbd304ab393d3f940d12af213"
I20260925 00:57:19.418540 3649 raft_consensus.cc:3037] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:19.418977 3780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57dbe03d30d14f708592ac42601472bf" candidate_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f86a1390a0243a38d283a6fee99b49d"
I20260925 00:57:19.419095 3780 raft_consensus.cc:3037] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:19.419587 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.415419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:59661 (local address 127.16.202.1:41979)
0925 00:57:19.415542 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.415547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.415560 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:57:19.415597 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.415599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.415646 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.415709 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.416333 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.417027 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.418381 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.418959 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.419038 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.419079 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.419415 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.419482 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:57:19.419683 3649 raft_consensus.cc:2454] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770df6ef0ded4aceb3a70e5b0f614efd in term 1.
I20260925 00:57:19.420024 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.414834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:43457 (local address 127.16.202.2:59757)
0925 00:57:19.415206 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.415220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.415297 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.415950 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.415953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.415969 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.416065 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.416071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.417644 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.417647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.418760 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.418770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.419631 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.419635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.419734 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.419755 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.419822 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.419891 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":16,"thread_start_us":118,"threads_started":1}
I20260925 00:57:19.420044 3452 leader_election.cc:302] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 002e144cbd304ab393d3f940d12af213, 770df6ef0ded4aceb3a70e5b0f614efd; no voters:
I20260925 00:57:19.419906 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.414618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:41979 (local address 127.16.202.2:59661)
0925 00:57:19.415174 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.415188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:19.415276 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:19.416009 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:19.416013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:19.416032 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:19.416157 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.416189 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.417147 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.417150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.418250 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:19.418260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.419184 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.419188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:19.419278 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.419300 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:57:19.419694 (+ 394us) client_negotiation.cc:240] Negotiation successful
0925 00:57:19.419769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":39}
I20260925 00:57:19.420148 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.415601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:59757 (local address 127.16.202.3:43457)
0925 00:57:19.415692 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.415695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.415707 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:57:19.415746 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.415749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.415790 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.415855 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:19.416871 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.417528 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.418881 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.419419 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.419529 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.419572 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.419974 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.420045 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:57:19.420338 3780 raft_consensus.cc:2454] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770df6ef0ded4aceb3a70e5b0f614efd in term 1.
I20260925 00:57:19.420707 3849 raft_consensus.cc:2781] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 1 FOLLOWER]: Leader election won for term 1
W20260925 00:57:19.420871 3827 tablet.cc:2369] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:19.420871 3518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57dbe03d30d14f708592ac42601472bf" candidate_uuid: "002e144cbd304ab393d3f940d12af213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" is_pre_election: true
I20260925 00:57:19.420946 3849 raft_consensus.cc:686] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 1 LEADER]: Becoming Leader. State: Replica: 770df6ef0ded4aceb3a70e5b0f614efd, State: Running, Role: LEADER
I20260925 00:57:19.421285 3849 consensus_queue.cc:237] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } }
I20260925 00:57:19.421427 3780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57dbe03d30d14f708592ac42601472bf" candidate_uuid: "002e144cbd304ab393d3f940d12af213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f86a1390a0243a38d283a6fee99b49d" is_pre_election: true
I20260925 00:57:19.421592 3780 raft_consensus.cc:2379] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 002e144cbd304ab393d3f940d12af213 in current term 1: Already voted for candidate 770df6ef0ded4aceb3a70e5b0f614efd in this term.
I20260925 00:57:19.423779 3584 leader_election.cc:302] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 002e144cbd304ab393d3f940d12af213; no voters: 5f86a1390a0243a38d283a6fee99b49d, 770df6ef0ded4aceb3a70e5b0f614efd
I20260925 00:57:19.424048 3851 raft_consensus.cc:2726] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 00:57:19.424605 3375 catalog_manager.cc:5495] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd reported cstate change: term changed from 0 to 1, leader changed from <none> to 770df6ef0ded4aceb3a70e5b0f614efd (127.16.202.1). New cstate: current_term: 1 leader_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770df6ef0ded4aceb3a70e5b0f614efd" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 41979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 } health_report { overall_health: UNKNOWN } } }
I20260925 00:57:19.450564 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.446659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.16.202.1:41979)
0925 00:57:19.446792 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:19.446795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:19.446810 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:57:19.446903 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:19.446905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:19.446959 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:19.447061 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:19.447314 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.447863 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.448737 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:19.448967 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:19.449169 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:19.449522 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:19.449976 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:19.450137 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:19.450328 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 00:57:19.450374 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:57:19.456907 3780 raft_consensus.cc:1261] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 1 FOLLOWER]: Refusing update from remote peer 770df6ef0ded4aceb3a70e5b0f614efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:19.456947 3649 raft_consensus.cc:1261] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 1 FOLLOWER]: Refusing update from remote peer 770df6ef0ded4aceb3a70e5b0f614efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:19.457614 3855 consensus_queue.cc:1035] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f86a1390a0243a38d283a6fee99b49d" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:19.457845 3849 consensus_queue.cc:1035] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "002e144cbd304ab393d3f940d12af213" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:19.466364 3860 mvcc.cc:205] Tried to move back new op lower bound from 7333059950407360512 to 7333059950271979520. Current Snapshot: MvccSnapshot[applied={T|T < 7333059950407360512}]
I20260925 00:57:19.468444 3859 mvcc.cc:205] Tried to move back new op lower bound from 7333059950407360512 to 7333059950271979520. Current Snapshot: MvccSnapshot[applied={T|T < 7333059950407360512}]
I20260925 00:57:19.474027 3375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:19.474264 3375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:19.476905 3375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e547994e8724941935e7f8f3f15af68: Sending DeleteTablet for 3 replicas of tablet 57dbe03d30d14f708592ac42601472bf
I20260925 00:57:19.477828 3760 tablet_service.cc:1514] Processing DeleteTablet for tablet 57dbe03d30d14f708592ac42601472bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:19 UTC) from {username='slave'} at 127.0.0.1:55546
I20260925 00:57:19.477821 3629 tablet_service.cc:1514] Processing DeleteTablet for tablet 57dbe03d30d14f708592ac42601472bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:19 UTC) from {username='slave'} at 127.0.0.1:58880
I20260925 00:57:19.477972 3498 tablet_service.cc:1514] Processing DeleteTablet for tablet 57dbe03d30d14f708592ac42601472bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:19 UTC) from {username='slave'} at 127.0.0.1:53704
I20260925 00:57:19.478526 3870 tablet_replica.cc:330] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd: stopping tablet replica
I20260925 00:57:19.478528 3869 tablet_replica.cc:330] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: stopping tablet replica
I20260925 00:57:19.478860 3869 raft_consensus.cc:2229] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:19.478860 3870 raft_consensus.cc:2229] T 57dbe03d30d14f708592ac42601472bf P 770df6ef0ded4aceb3a70e5b0f614efd [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:57:19.478955 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3436
I20260925 00:57:19.479200 3869 raft_consensus.cc:2258] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:19.479601 3868 tablet_replica.cc:330] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: stopping tablet replica
I20260925 00:57:19.479914 3868 raft_consensus.cc:2229] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:19.480208 3868 raft_consensus.cc:2258] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:19.481750 3868 ts_tablet_manager.cc:1905] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:19.484562 3868 ts_tablet_manager.cc:1918] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:19.484660 3868 log.cc:1196] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/57dbe03d30d14f708592ac42601472bf
I20260925 00:57:19.485001 3868 ts_tablet_manager.cc:1939] T 57dbe03d30d14f708592ac42601472bf P 002e144cbd304ab393d3f940d12af213: Deleting consensus metadata
I20260925 00:57:19.485719 3869 ts_tablet_manager.cc:1905] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:19.486304 3361 catalog_manager.cc:4841] TS 002e144cbd304ab393d3f940d12af213 (127.16.202.2:44251): tablet 57dbe03d30d14f708592ac42601472bf (table test-table [id=6640faf9d000475cb496aada97147097]) successfully deleted
W20260925 00:57:19.486984 3362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:41979 (error 108)
I20260925 00:57:19.487089 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3567
I20260925 00:57:19.488659 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:19.488325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:41979 (local address 127.0.0.1:53728)
0925 00:57:19.488509 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:19.488574 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 00:57:19.488801 3869 ts_tablet_manager.cc:1918] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 00:57:19.488847 3362 catalog_manager.cc:4584] TS 770df6ef0ded4aceb3a70e5b0f614efd (127.16.202.1:41979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57dbe03d30d14f708592ac42601472bf: Network error: Client connection negotiation failed: client connection to 127.16.202.1:41979: connect: Connection refused (error 111)
I20260925 00:57:19.488896 3869 log.cc:1196] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/57dbe03d30d14f708592ac42601472bf
I20260925 00:57:19.489133 3869 ts_tablet_manager.cc:1939] T 57dbe03d30d14f708592ac42601472bf P 5f86a1390a0243a38d283a6fee99b49d: Deleting consensus metadata
I20260925 00:57:19.490164 3362 catalog_manager.cc:4841] TS 5f86a1390a0243a38d283a6fee99b49d (127.16.202.3:43457): tablet 57dbe03d30d14f708592ac42601472bf (table test-table [id=6640faf9d000475cb496aada97147097]) successfully deleted
I20260925 00:57:19.493160 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3698
I20260925 00:57:19.497669 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3345
2026-09-25T00:57:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1183 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:19 dist-test-slave-rpk1 krb5kdc[3874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:19 dist-test-slave-rpk1 krb5kdc[3874](info): set up 2 sockets
Sep 25 00:57:19 dist-test-slave-rpk1 krb5kdc[3874](info): commencing operation
krb5kdc: starting...
W20260925 00:57:20.456709 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:20 dist-test-slave-rpk1 krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:20.580519 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38215
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:20.691184 3887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:20.691486 3887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:20.691543 3887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:20.695721 3887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:20.695806 3887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:20.695833 3887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:20.695861 3887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:20.695883 3887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:20.701470 3887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:20.702584 3887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:20.703629 3887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:20.712114 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:20.712100 3895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:20.712193 3887 server_base.cc:1034] running on GCE node
W20260925 00:57:20.712105 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:20.713024 3887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:20.714316 3887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:20.715516 3887 hybrid_clock.cc:648] HybridClock initialized: now 1790297840715493 us; error 47 us; skew 500 ppm
Sep 25 00:57:20 dist-test-slave-rpk1 krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297840, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:20.718642 3887 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:20.719617 3887 webserver.cc:460] Webserver started at http://127.16.202.62:43453/ using document root <none> and password file <none>
I20260925 00:57:20.720172 3887 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:20.720233 3887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:20.720404 3887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:20.722579 3887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "6f11063684cf473d9c6c3d05751c9399"
format_stamp: "Formatted at 2026-09-25 00:57:20 on dist-test-slave-rpk1"
server_key: "51af62bcf7425266cbe4490997450539"
server_key_iv: "b13d715d9f251819eebf75edc4deff89"
server_key_version: "encryptionkey@0"
I20260925 00:57:20.723109 3887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "6f11063684cf473d9c6c3d05751c9399"
format_stamp: "Formatted at 2026-09-25 00:57:20 on dist-test-slave-rpk1"
server_key: "51af62bcf7425266cbe4490997450539"
server_key_iv: "b13d715d9f251819eebf75edc4deff89"
server_key_version: "encryptionkey@0"
I20260925 00:57:20.726979 3887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:20.729453 3902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:20.730679 3887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:20.730793 3887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "6f11063684cf473d9c6c3d05751c9399"
format_stamp: "Formatted at 2026-09-25 00:57:20 on dist-test-slave-rpk1"
server_key: "51af62bcf7425266cbe4490997450539"
server_key_iv: "b13d715d9f251819eebf75edc4deff89"
server_key_version: "encryptionkey@0"
I20260925 00:57:20.730887 3887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:20.741770 3887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:20.745201 3887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:20.745365 3887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:20.752230 3887 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:38215
I20260925 00:57:20.752265 3954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:38215 every 8 connection(s)
I20260925 00:57:20.753226 3887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:20.756237 3955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:20.757117 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3887
I20260925 00:57:20.757227 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:20.757514 17192 external_mini_cluster.cc:1428] Setting key 7b854896dd68784ce1ce6323bd6f2f13
I20260925 00:57:20.762933 3955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399: Bootstrap starting.
I20260925 00:57:20.765775 3955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:20.766695 3955 log.cc:824] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:57:20 dist-test-slave-rpk1 krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:20.768704 3955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399: No bootstrap required, opened a new log
I20260925 00:57:20.772526 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:20.758921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.16.202.62:38215)
0925 00:57:20.759358 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:20.759375 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:20.759421 (+ 46us) server_negotiation.cc:407] Connection header received
0925 00:57:20.760077 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:20.760095 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:20.760460 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:20.760835 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:20.761726 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:20.762797 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:20.763955 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:20.764508 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:20.764638 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:20.767466 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:20.767473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:20.767489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:20.767520 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:20.769234 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:20.769998 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:20.770003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:20.770007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:20.770122 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:20.770523 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:20.770526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:20.770527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:20.770962 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:20.771113 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:20.771463 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 00:57:20.771770 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260925 00:57:20.772902 3955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f11063684cf473d9c6c3d05751c9399" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38215 } }
I20260925 00:57:20.773139 3955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:20.773203 3955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f11063684cf473d9c6c3d05751c9399, State: Initialized, Role: FOLLOWER
I20260925 00:57:20.773646 3955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f11063684cf473d9c6c3d05751c9399" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38215 } }
I20260925 00:57:20.773805 3955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:20.773878 3955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:20.773968 3955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:20.775051 3955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f11063684cf473d9c6c3d05751c9399" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38215 } }
I20260925 00:57:20.775352 3955 leader_election.cc:302] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f11063684cf473d9c6c3d05751c9399; no voters:
I20260925 00:57:20.775615 3955 leader_election.cc:288] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:20.775728 3960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:20.775919 3960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [term 1 LEADER]: Becoming Leader. State: Replica: 6f11063684cf473d9c6c3d05751c9399, State: Running, Role: LEADER
I20260925 00:57:20.776233 3960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f11063684cf473d9c6c3d05751c9399" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38215 } }
I20260925 00:57:20.776788 3955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:20.777846 3962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f11063684cf473d9c6c3d05751c9399. Latest consensus state: current_term: 1 leader_uuid: "6f11063684cf473d9c6c3d05751c9399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f11063684cf473d9c6c3d05751c9399" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38215 } } }
I20260925 00:57:20.777982 3962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:20.778282 3961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f11063684cf473d9c6c3d05751c9399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f11063684cf473d9c6c3d05751c9399" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38215 } } }
I20260925 00:57:20.778371 3961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:20.778517 3969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:20.781347 3969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:20.786767 3969 catalog_manager.cc:1347] Generated new cluster ID: b7c48ea78a2a4963b2ed72564024f298
I20260925 00:57:20.786846 3969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:20.798810 3969 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:20.799815 3969 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:20.804944 3969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399: Generated new TSK 0
I20260925 00:57:20.805620 3969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.202.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
Entry for principal kudu/127.16.202.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab.
I20260925 00:57:20.833770 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38215
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:20.946861 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:20.947158 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:20.947204 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:20.951315 3981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:20.951391 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:20.951453 3981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:57:20.957283 3981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.1.keytab
--principal=kudu/127.16.202.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:20.958351 3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:20.959350 3981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:20.969532 3986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:20.969691 3987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:20.969801 3989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:20.969713 3981 server_base.cc:1034] running on GCE node
I20260925 00:57:20.970443 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:20.971168 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:20.972407 3981 hybrid_clock.cc:648] HybridClock initialized: now 1790297840972340 us; error 37 us; skew 500 ppm
Sep 25 00:57:20 dist-test-slave-rpk1 krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297840, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:20.975421 3981 init.cc:377] Logged in from keytab as kudu/127.16.202.1@KRBTEST.COM (short username kudu)
I20260925 00:57:20.976418 3981 webserver.cc:460] Webserver started at http://127.16.202.1:43615/ using document root <none> and password file <none>
I20260925 00:57:20.977044 3981 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:20.977124 3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:20.977342 3981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:20.979564 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "d497336bbbe34cccb743cc72b1d7c105"
format_stamp: "Formatted at 2026-09-25 00:57:20 on dist-test-slave-rpk1"
server_key: "bcba4e891290a19b9bdadf3524b44297"
server_key_iv: "e90ed817f671789f1e4818fe94ff0146"
server_key_version: "encryptionkey@0"
I20260925 00:57:20.980165 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "d497336bbbe34cccb743cc72b1d7c105"
format_stamp: "Formatted at 2026-09-25 00:57:20 on dist-test-slave-rpk1"
server_key: "bcba4e891290a19b9bdadf3524b44297"
server_key_iv: "e90ed817f671789f1e4818fe94ff0146"
server_key_version: "encryptionkey@0"
I20260925 00:57:20.984133 3981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:20.986886 3996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:20.988116 3981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:57:20.988231 3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "d497336bbbe34cccb743cc72b1d7c105"
format_stamp: "Formatted at 2026-09-25 00:57:20 on dist-test-slave-rpk1"
server_key: "bcba4e891290a19b9bdadf3524b44297"
server_key_iv: "e90ed817f671789f1e4818fe94ff0146"
server_key_version: "encryptionkey@0"
I20260925 00:57:20.988336 3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:21.001752 3981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:21.005008 3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:21.005227 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:21.005906 3981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:21.006768 3981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:21.006816 3981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.006868 3981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:21.006897 3981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.016176 3981 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:44305
I20260925 00:57:21.016228 4109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:44305 every 8 connection(s)
I20260925 00:57:21.017195 3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:57:21.020726 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 3981
I20260925 00:57:21.020828 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:57:21.021095 17192 external_mini_cluster.cc:1428] Setting key 969064a338ba8bb1b1f0f51f0e9e68bd
Sep 25 00:57:21 dist-test-slave-rpk1 krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297840, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.1@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.202.2@KRBTEST.COM; defaulting to no policy
I20260925 00:57:21.031342 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.019249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:34049 (local address 127.16.202.62:38215)
0925 00:57:21.019418 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.019422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.020218 (+ 796us) server_negotiation.cc:407] Connection header received
0925 00:57:21.021695 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.021698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.021755 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.021839 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:21.023111 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.023642 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.024801 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.025061 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.025149 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.027634 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:21.027637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:21.027639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:21.027663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:21.029107 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.029626 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.029630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.029631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.029709 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.030117 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.030120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.030121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.030316 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:21.030447 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.031034 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.031172 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:57:21.032083 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.019618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38215 (local address 127.16.202.1:34049)
0925 00:57:21.020066 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.020110 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.021479 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.022008 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.022015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.022445 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:21.022964 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.022978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.023761 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.023764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.024646 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.024655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.025201 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.025205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.025384 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.025957 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:21.025975 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:21.027469 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:21.029226 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:21.029233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:21.029250 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:21.029527 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:21.029810 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:21.029814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:21.029816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:21.030007 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:21.030488 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:21.030494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:21.030770 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.030938 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.031234 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.2@KRBTEST.COM" created.
I20260925 00:57:21.033429 4110 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38215
I20260925 00:57:21.033666 4110 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:21.034257 4110 heartbeater.cc:511] Master 127.16.202.62:38215 requested a full tablet report, sending...
I20260925 00:57:21.035939 3919 ts_manager.cc:194] Registered new tserver with Master: d497336bbbe34cccb743cc72b1d7c105 (127.16.202.1:44305)
I20260925 00:57:21.037619 3919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.1@KRBTEST.COM'} at 127.16.202.1:34049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
Entry for principal kudu/127.16.202.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab.
I20260925 00:57:21.048343 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38215
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:21.158685 4115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:21.158987 4115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:21.159032 4115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:21.163160 4115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:21.163244 4115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:21.163303 4115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:57:21.168958 4115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.2.keytab
--principal=kudu/127.16.202.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:21.170089 4115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:21.171088 4115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:57:21.180770 4115 server_base.cc:1034] running on GCE node
W20260925 00:57:21.180709 4120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:21.180791 4123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:21.180707 4121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:21.181421 4115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:21.182106 4115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:21.183285 4115 hybrid_clock.cc:648] HybridClock initialized: now 1790297841183238 us; error 65 us; skew 500 ppm
Sep 25 00:57:21 dist-test-slave-rpk1 krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297841, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:21.186537 4115 init.cc:377] Logged in from keytab as kudu/127.16.202.2@KRBTEST.COM (short username kudu)
I20260925 00:57:21.187561 4115 webserver.cc:460] Webserver started at http://127.16.202.2:44043/ using document root <none> and password file <none>
I20260925 00:57:21.188133 4115 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:21.188206 4115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:21.188426 4115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:21.190665 4115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "cef9fc60ca254abdae6016cddef8605d"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "fab403947f5b52b3c15b1b56e0289a5a"
server_key_iv: "707af2e4787939cdd40dd4f91545b060"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.191250 4115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "cef9fc60ca254abdae6016cddef8605d"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "fab403947f5b52b3c15b1b56e0289a5a"
server_key_iv: "707af2e4787939cdd40dd4f91545b060"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.195364 4115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:57:21.198122 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.199460 4115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:57:21.199590 4115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "cef9fc60ca254abdae6016cddef8605d"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "fab403947f5b52b3c15b1b56e0289a5a"
server_key_iv: "707af2e4787939cdd40dd4f91545b060"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.199697 4115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:21.214955 4115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:21.218029 4115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:21.218218 4115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:21.218928 4115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:21.219894 4115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:21.219946 4115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.219987 4115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:21.220007 4115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.228600 4115 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:44085
I20260925 00:57:21.228631 4243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:44085 every 8 connection(s)
I20260925 00:57:21.229549 4115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:57:21.235059 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4115
I20260925 00:57:21.235183 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:57:21.235532 17192 external_mini_cluster.cc:1428] Setting key d09e29be55717899eb71317cca02b070
Sep 25 00:57:21 dist-test-slave-rpk1 krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297841, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.2@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:21.245012 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.231485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:56649 (local address 127.16.202.62:38215)
0925 00:57:21.231636 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.231640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.232497 (+ 857us) server_negotiation.cc:407] Connection header received
0925 00:57:21.233979 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.233982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.234031 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.234104 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:21.235630 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.236326 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.237845 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.238154 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.238265 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.241058 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:21.241062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:21.241064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:21.241093 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:21.242751 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.243287 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.243290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.243292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.243389 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.243740 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.243743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.243745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.243950 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:21.244039 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.244657 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.244830 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:57:21.246013 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.231842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38215 (local address 127.16.202.2:56649)
0925 00:57:21.232338 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.232388 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.233737 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.234316 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.234326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.234727 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:21.235456 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.235479 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.236456 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.236459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.237685 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.237697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.238304 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.238307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.238482 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.239116 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:21.239137 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:21.240877 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:21.242878 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:21.242883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:21.242898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:21.243167 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:21.243477 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:21.243480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:21.243482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:21.243635 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:21.244049 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:21.244057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:21.244350 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.244588 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.244974 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.16.202.3@KRBTEST.COM; defaulting to no policy
I20260925 00:57:21.247316 4244 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38215
I20260925 00:57:21.247602 4244 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:21.248145 4244 heartbeater.cc:511] Master 127.16.202.62:38215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.3@KRBTEST.COM" created.
I20260925 00:57:21.249423 3919 ts_manager.cc:194] Registered new tserver with Master: cef9fc60ca254abdae6016cddef8605d (127.16.202.2:44085)
I20260925 00:57:21.250226 3919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.2@KRBTEST.COM'} at 127.16.202.2:56649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
Entry for principal kudu/127.16.202.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab.
I20260925 00:57:21.265694 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38215
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:21.375751 4249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:21.376035 4249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:21.376078 4249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:21.380641 4249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:21.380733 4249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:21.380807 4249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:57:21.386648 4249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.3.keytab
--principal=kudu/127.16.202.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:21.387897 4249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:21.389034 4249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:21.398802 4255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:21.398805 4254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:21.398854 4257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:21.399168 4249 server_base.cc:1034] running on GCE node
I20260925 00:57:21.399580 4249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:21.400219 4249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:21.401398 4249 hybrid_clock.cc:648] HybridClock initialized: now 1790297841401337 us; error 80 us; skew 500 ppm
Sep 25 00:57:21 dist-test-slave-rpk1 krb5kdc[3874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297841, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:21.404297 4249 init.cc:377] Logged in from keytab as kudu/127.16.202.3@KRBTEST.COM (short username kudu)
I20260925 00:57:21.405227 4249 webserver.cc:460] Webserver started at http://127.16.202.3:46423/ using document root <none> and password file <none>
I20260925 00:57:21.405772 4249 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:21.405845 4249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:21.406044 4249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:21.408232 4249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "b24d5bf34820480d81bfc11d7d04f473"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "6ad6f634f576395573c4a1ed7b53d4ac"
server_key_iv: "f1b3b348bc381f5f3512e691a4865208"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.408849 4249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "b24d5bf34820480d81bfc11d7d04f473"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "6ad6f634f576395573c4a1ed7b53d4ac"
server_key_iv: "f1b3b348bc381f5f3512e691a4865208"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.412812 4249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:21.415309 4264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.416538 4249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:57:21.416646 4249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "b24d5bf34820480d81bfc11d7d04f473"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "6ad6f634f576395573c4a1ed7b53d4ac"
server_key_iv: "f1b3b348bc381f5f3512e691a4865208"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.416780 4249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:21.440326 4249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:21.443939 4249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:21.444149 4249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:21.444809 4249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:21.445704 4249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:21.445755 4249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.445807 4249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:21.445837 4249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.454425 4249 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:34089
I20260925 00:57:21.454476 4377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:34089 every 8 connection(s)
I20260925 00:57:21.455345 4249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:57:21.462428 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4249
I20260925 00:57:21.462558 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:57:21.462831 17192 external_mini_cluster.cc:1428] Setting key 40fcdc1edf5c137f59ee8bc75179fe86
Sep 25 00:57:21 dist-test-slave-rpk1 krb5kdc[3874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297841, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.3@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:21.469300 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.457246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:50709 (local address 127.16.202.62:38215)
0925 00:57:21.457466 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.457471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.458244 (+ 773us) server_negotiation.cc:407] Connection header received
0925 00:57:21.459620 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.459623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.459674 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.459753 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:21.461028 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.461585 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.463044 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.463276 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.463418 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.466060 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:21.466063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:21.466065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:21.466091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:21.467418 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.467896 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.467899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.467900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.467979 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.468255 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.468257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.468258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.468471 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:21.468558 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.469074 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.469176 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:57:21.470161 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.457636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:38215 (local address 127.16.202.3:50709)
0925 00:57:21.458092 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.458136 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.459366 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.459926 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.459934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.460303 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:21.460854 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.460879 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.461713 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.461716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.462921 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.462933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.463482 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.463487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.463668 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.464330 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:21.464350 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:21.465894 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:21.467526 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:21.467532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:21.467547 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:21.467806 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:21.468050 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:21.468052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:21.468054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:21.468189 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:21.468552 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:21.468557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:21.468855 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.469035 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.469299 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20260925 00:57:21.471282 4378 heartbeater.cc:347] Connected to a master server at 127.16.202.62:38215
I20260925 00:57:21.471541 4378 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:21.471999 4378 heartbeater.cc:511] Master 127.16.202.62:38215 requested a full tablet report, sending...
I20260925 00:57:21.473045 3919 ts_manager.cc:194] Registered new tserver with Master: b24d5bf34820480d81bfc11d7d04f473 (127.16.202.3:34089)
I20260925 00:57:21.473606 3919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.202.3@KRBTEST.COM'} at 127.16.202.3:50709
I20260925 00:57:21.477128 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:57:21.488260 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.479232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.16.202.62:38215)
0925 00:57:21.479354 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.479359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.479600 (+ 241us) server_negotiation.cc:407] Connection header received
0925 00:57:21.479758 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.479761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.479809 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.479895 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:21.480713 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.481515 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.482695 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.483048 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.483121 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.484784 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:21.484788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:21.484790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:21.484817 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:21.486400 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.487038 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.487042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.487044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.487128 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:21.487393 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:21.487396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:21.487398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:21.487630 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:21.487754 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.487989 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.488103 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:57:21.491528 3919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:57:21.494728 3919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:57:21.509466 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.504113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.16.202.3:34089)
0925 00:57:21.504537 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.504549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.504805 (+ 256us) server_negotiation.cc:407] Connection header received
0925 00:57:21.504888 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.504893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.505099 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.505244 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:21.505705 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.506792 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.508076 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.508770 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.508872 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.509047 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.509201 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.509302 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":150,"threads_started":1}
I20260925 00:57:21.509462 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.503570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:34089 (local address 127.0.0.1:50698)
0925 00:57:21.504617 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.504630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.504749 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.505422 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.505425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.505443 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:21.505584 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.505591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.506929 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.506933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.507949 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.507959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.508887 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.508891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.509010 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.509049 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.509157 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.509302 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":19,"thread_start_us":72,"threads_started":1}
I20260925 00:57:21.509505 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.503285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44305 (local address 127.0.0.1:48168)
0925 00:57:21.504582 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.504593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.504663 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.505180 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.505186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.505208 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:21.505362 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.505369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.506783 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.506786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.507945 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.507956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.509104 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.509108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.509208 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.509232 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.509278 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.509354 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":4,"spinlock_wait_cycles":8448,"thread_start_us":99,"threads_started":1}
I20260925 00:57:21.510290 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.504119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.16.202.1:44305)
0925 00:57:21.504737 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.504752 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.504777 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:57:21.504861 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.504866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.505042 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.505179 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:21.505469 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.506641 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.508073 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.508969 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.509853 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.509972 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.510043 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.510137 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":70,"threads_started":1}
I20260925 00:57:21.512275 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.503257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:44085 (local address 127.0.0.1:36560)
0925 00:57:21.504128 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.504183 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.504312 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.505457 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.505459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.505473 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:21.505598 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.505603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.509660 (+ 4057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.509664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.510716 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.510727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.511810 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.511814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.511947 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.511972 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.512053 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.512133 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":23,"thread_start_us":110,"threads_started":1}
I20260925 00:57:21.512338 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.503362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.16.202.2:44085)
0925 00:57:21.504928 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.504939 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.504958 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:57:21.505019 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.505023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.505185 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.505308 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:21.508105 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.509260 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.510863 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.511675 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.511786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.511924 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.512091 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.512189 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":109,"threads_started":1}
I20260925 00:57:21.513099 4312 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b (DEFAULT_TABLE table=test-table [id=baea25e1b3714f46b6b587db2961cbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:21.513103 4044 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b (DEFAULT_TABLE table=test-table [id=baea25e1b3714f46b6b587db2961cbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:21.514070 4312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:21.514084 4044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:21.520735 4395 tablet_bootstrap.cc:492] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Bootstrap starting.
I20260925 00:57:21.520593 4178 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b (DEFAULT_TABLE table=test-table [id=baea25e1b3714f46b6b587db2961cbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:21.521818 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:21.522073 4396 tablet_bootstrap.cc:492] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105: Bootstrap starting.
I20260925 00:57:21.522939 4395 tablet_bootstrap.cc:654] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:21.523936 4395 log.cc:824] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:21.524235 4396 tablet_bootstrap.cc:654] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:21.525563 4396 log.cc:824] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:21.525943 4395 tablet_bootstrap.cc:492] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: No bootstrap required, opened a new log
I20260925 00:57:21.526151 4395 ts_tablet_manager.cc:1397] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:57:21.527730 4396 tablet_bootstrap.cc:492] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105: No bootstrap required, opened a new log
I20260925 00:57:21.527935 4396 ts_tablet_manager.cc:1397] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:57:21.529089 4395 raft_consensus.cc:348] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.529340 4395 raft_consensus.cc:374] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:21.529400 4395 raft_consensus.cc:729] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24d5bf34820480d81bfc11d7d04f473, State: Initialized, Role: FOLLOWER
I20260925 00:57:21.529891 4399 tablet_bootstrap.cc:492] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: Bootstrap starting.
I20260925 00:57:21.529877 4395 consensus_queue.cc:260] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.530936 4395 ts_tablet_manager.cc:1428] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260925 00:57:21.531004 4378 heartbeater.cc:503] Master 127.16.202.62:38215 was elected leader, sending a full tablet report...
I20260925 00:57:21.531754 4396 raft_consensus.cc:348] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.532112 4396 raft_consensus.cc:374] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:21.532224 4396 raft_consensus.cc:729] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d497336bbbe34cccb743cc72b1d7c105, State: Initialized, Role: FOLLOWER
I20260925 00:57:21.532794 4396 consensus_queue.cc:260] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.533692 4399 tablet_bootstrap.cc:654] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:21.533756 4396 ts_tablet_manager.cc:1428] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:57:21.534379 4110 heartbeater.cc:503] Master 127.16.202.62:38215 was elected leader, sending a full tablet report...
I20260925 00:57:21.534680 4399 log.cc:824] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:21.536969 4399 tablet_bootstrap.cc:492] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: No bootstrap required, opened a new log
I20260925 00:57:21.537168 4399 ts_tablet_manager.cc:1397] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:57:21.540169 4399 raft_consensus.cc:348] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.540447 4399 raft_consensus.cc:374] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:21.540496 4399 raft_consensus.cc:729] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef9fc60ca254abdae6016cddef8605d, State: Initialized, Role: FOLLOWER
I20260925 00:57:21.541049 4399 consensus_queue.cc:260] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.541795 4244 heartbeater.cc:503] Master 127.16.202.62:38215 was elected leader, sending a full tablet report...
I20260925 00:57:21.543082 4399 ts_tablet_manager.cc:1428] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:57:21.696482 4400 raft_consensus.cc:482] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:21.696753 4400 raft_consensus.cc:504] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.697932 4400 leader_election.cc:288] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef9fc60ca254abdae6016cddef8605d (127.16.202.2:44085), d497336bbbe34cccb743cc72b1d7c105 (127.16.202.1:44305)
I20260925 00:57:21.701488 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.698343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:37179 (local address 127.16.202.1:44305)
0925 00:57:21.698492 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.698497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.698514 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:57:21.698676 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.698679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.698731 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.698792 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:21.699043 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.699530 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.700526 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.700973 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.701080 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.701135 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.701291 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.701358 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:57:21.701488 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.698237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44305 (local address 127.16.202.3:37179)
0925 00:57:21.698421 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.698437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.698549 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.698816 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.698819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.698835 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:21.698941 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.698947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.699628 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.699631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.700408 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.700417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.701081 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.701084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.701189 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.701207 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.701270 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.701355 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 00:57:21.701665 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.698248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:44085 (local address 127.16.202.3:32769)
0925 00:57:21.698689 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.698699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:21.698798 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:21.699032 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:21.699034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:21.699050 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:21.699165 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.699171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.699849 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.699853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.700640 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:21.700649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.701256 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.701259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:21.701374 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.701392 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:57:21.701460 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 00:57:21.701526 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":37,"thread_start_us":118,"threads_started":1}
I20260925 00:57:21.701654 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.698496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:32769 (local address 127.16.202.2:44085)
0925 00:57:21.698625 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.698630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.698773 (+ 143us) server_negotiation.cc:407] Connection header received
0925 00:57:21.698895 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.698897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.698947 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.699008 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:21.699270 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.699750 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.700775 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.701155 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.701256 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.701305 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.701486 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.701555 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:57:21.702291 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbb7971e6eb4e39abdd89afaa58eb4b" candidate_uuid: "b24d5bf34820480d81bfc11d7d04f473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef9fc60ca254abdae6016cddef8605d" is_pre_election: true
I20260925 00:57:21.702365 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbb7971e6eb4e39abdd89afaa58eb4b" candidate_uuid: "b24d5bf34820480d81bfc11d7d04f473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d497336bbbe34cccb743cc72b1d7c105" is_pre_election: true
I20260925 00:57:21.702621 4198 raft_consensus.cc:2454] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24d5bf34820480d81bfc11d7d04f473 in term 0.
I20260925 00:57:21.702752 4064 raft_consensus.cc:2454] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24d5bf34820480d81bfc11d7d04f473 in term 0.
I20260925 00:57:21.703167 4266 leader_election.cc:302] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24d5bf34820480d81bfc11d7d04f473, cef9fc60ca254abdae6016cddef8605d; no voters:
I20260925 00:57:21.703460 4400 raft_consensus.cc:2781] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:57:21.703545 4400 raft_consensus.cc:482] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:57:21.703578 4400 raft_consensus.cc:3037] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:21.704698 4400 raft_consensus.cc:504] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.705066 4400 leader_election.cc:288] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [CANDIDATE]: Term 1 election: Requested vote from peers cef9fc60ca254abdae6016cddef8605d (127.16.202.2:44085), d497336bbbe34cccb743cc72b1d7c105 (127.16.202.1:44305)
I20260925 00:57:21.705458 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbb7971e6eb4e39abdd89afaa58eb4b" candidate_uuid: "b24d5bf34820480d81bfc11d7d04f473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d497336bbbe34cccb743cc72b1d7c105"
I20260925 00:57:21.705458 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbb7971e6eb4e39abdd89afaa58eb4b" candidate_uuid: "b24d5bf34820480d81bfc11d7d04f473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef9fc60ca254abdae6016cddef8605d"
I20260925 00:57:21.705600 4064 raft_consensus.cc:3037] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:21.705600 4198 raft_consensus.cc:3037] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 0 FOLLOWER]: Advancing to term 1
W20260925 00:57:21.706375 4379 tablet.cc:2369] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:21.706907 4198 raft_consensus.cc:2454] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24d5bf34820480d81bfc11d7d04f473 in term 1.
I20260925 00:57:21.706907 4064 raft_consensus.cc:2454] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24d5bf34820480d81bfc11d7d04f473 in term 1.
I20260925 00:57:21.707237 4266 leader_election.cc:302] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24d5bf34820480d81bfc11d7d04f473, cef9fc60ca254abdae6016cddef8605d; no voters:
I20260925 00:57:21.707494 4400 raft_consensus.cc:2781] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:21.707706 4400 raft_consensus.cc:686] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 1 LEADER]: Becoming Leader. State: Replica: b24d5bf34820480d81bfc11d7d04f473, State: Running, Role: LEADER
I20260925 00:57:21.708050 4400 consensus_queue.cc:237] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } }
I20260925 00:57:21.711293 3919 catalog_manager.cc:5495] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 reported cstate change: term changed from 0 to 1, leader changed from <none> to b24d5bf34820480d81bfc11d7d04f473 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "b24d5bf34820480d81bfc11d7d04f473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b24d5bf34820480d81bfc11d7d04f473" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34089 } health_report { overall_health: HEALTHY } } }
W20260925 00:57:21.730711 4245 tablet.cc:2369] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:21.746080 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.742554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.16.202.3:34089)
0925 00:57:21.742689 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:21.742693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:21.742707 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:57:21.742785 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:21.742787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:21.742839 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:21.742939 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:21.743181 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.743701 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.744541 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:21.744776 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:21.744880 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:21.745250 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:21.745684 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:21.745780 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:21.745890 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 00:57:21.745945 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:21.752149 4198 raft_consensus.cc:1261] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 1 FOLLOWER]: Refusing update from remote peer b24d5bf34820480d81bfc11d7d04f473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:21.752149 4064 raft_consensus.cc:1261] T 9bbb7971e6eb4e39abdd89afaa58eb4b P d497336bbbe34cccb743cc72b1d7c105 [term 1 FOLLOWER]: Refusing update from remote peer b24d5bf34820480d81bfc11d7d04f473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:21.752960 4405 consensus_queue.cc:1035] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d497336bbbe34cccb743cc72b1d7c105" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:21.753157 4400 consensus_queue.cc:1035] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef9fc60ca254abdae6016cddef8605d" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:21.761878 4410 mvcc.cc:205] Tried to move back new op lower bound from 7333059959809028096 to 7333059959638970368. Current Snapshot: MvccSnapshot[applied={T|T < 7333059959809028096}]
I20260925 00:57:21.762141 4409 mvcc.cc:205] Tried to move back new op lower bound from 7333059959809028096 to 7333059959638970368. Current Snapshot: MvccSnapshot[applied={T|T < 7333059959809028096}]
I20260925 00:57:21.762866 4408 mvcc.cc:205] Tried to move back new op lower bound from 7333059959809028096 to 7333059959638970368. Current Snapshot: MvccSnapshot[applied={T|T < 7333059959809028096 or (T in {7333059959809028096})}]
I20260925 00:57:21.770045 3919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:21.770238 3919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:21.772825 3919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f11063684cf473d9c6c3d05751c9399: Sending DeleteTablet for 3 replicas of tablet 9bbb7971e6eb4e39abdd89afaa58eb4b
I20260925 00:57:21.773664 4044 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:21 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:48168
I20260925 00:57:21.773666 4178 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:21 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:36560
I20260925 00:57:21.773751 4312 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:21 UTC) from {username='kudu', principal='kudu/127.16.202.62@KRBTEST.COM'} at 127.0.0.1:50698
I20260925 00:57:21.774272 4418 tablet_replica.cc:330] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d: stopping tablet replica
I20260925 00:57:21.774369 4419 tablet_replica.cc:330] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: stopping tablet replica
I20260925 00:57:21.774669 4419 raft_consensus.cc:2229] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:57:21.774672 4418 raft_consensus.cc:2229] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:21.774727 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3981
I20260925 00:57:21.775023 4418 raft_consensus.cc:2258] T 9bbb7971e6eb4e39abdd89afaa58eb4b P cef9fc60ca254abdae6016cddef8605d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:21.775061 4419 raft_consensus.cc:2258] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:21.777161 4419 ts_tablet_manager.cc:1905] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:21.780296 4419 ts_tablet_manager.cc:1918] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:21.780401 4419 log.cc:1196] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/9bbb7971e6eb4e39abdd89afaa58eb4b
I20260925 00:57:21.780721 4419 ts_tablet_manager.cc:1939] T 9bbb7971e6eb4e39abdd89afaa58eb4b P b24d5bf34820480d81bfc11d7d04f473: Deleting consensus metadata
W20260925 00:57:21.781214 3905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:44305 (error 108)
I20260925 00:57:21.781381 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4115
I20260925 00:57:21.782773 3906 catalog_manager.cc:4841] TS b24d5bf34820480d81bfc11d7d04f473 (127.16.202.3:34089): tablet 9bbb7971e6eb4e39abdd89afaa58eb4b (table test-table [id=baea25e1b3714f46b6b587db2961cbb7]) successfully deleted
I20260925 00:57:21.785724 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.785367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:44305 (local address 127.0.0.1:48180)
0925 00:57:21.785523 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.785616 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 00:57:21.785966 3905 catalog_manager.cc:4584] TS d497336bbbe34cccb743cc72b1d7c105 (127.16.202.1:44305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bbb7971e6eb4e39abdd89afaa58eb4b: Network error: Client connection negotiation failed: client connection to 127.16.202.1:44305: connect: Connection refused (error 111)
I20260925 00:57:21.787118 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4249
I20260925 00:57:21.788126 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:21.787320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:44085 (local address 127.0.0.1:36564)
0925 00:57:21.787996 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:21.788049 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.2:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583}
I20260925 00:57:21.792737 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 3887
2026-09-25T00:57:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2296 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T00:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:21Z Disabled control of system clock
I20260925 00:57:21.821705 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40031
--webserver_interface=127.16.202.62
--webserver_port=0
--builtin_ntp_servers=127.16.202.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:57:21.932564 4424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:21.932883 4424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:21.932921 4424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:21.937109 4424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:21.937184 4424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:21.937198 4424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:21.937214 4424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:21.937229 4424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:21.942233 4424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:40031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:21.943434 4424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:21.944886 4424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:21.954260 4432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:21.954263 4429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:21.954263 4430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:21.954515 4424 server_base.cc:1034] running on GCE node
I20260925 00:57:21.955188 4424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:21.956419 4424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:21.957626 4424 hybrid_clock.cc:648] HybridClock initialized: now 1790297841957597 us; error 45 us; skew 500 ppm
I20260925 00:57:21.959785 4424 webserver.cc:460] Webserver started at http://127.16.202.62:33799/ using document root <none> and password file <none>
I20260925 00:57:21.960448 4424 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:21.960533 4424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:21.960822 4424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:21.963147 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "396c09392c5549a083b808fc4a553da2"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "8de14074c477b3a09904c3b70f0ba15a"
server_key_iv: "ceafcfb5aa37320818d24a2045b96cf9"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.963804 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "396c09392c5549a083b808fc4a553da2"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "8de14074c477b3a09904c3b70f0ba15a"
server_key_iv: "ceafcfb5aa37320818d24a2045b96cf9"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.968055 4424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:21.970748 4438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:21.972246 4424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 00:57:21.972376 4424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "396c09392c5549a083b808fc4a553da2"
format_stamp: "Formatted at 2026-09-25 00:57:21 on dist-test-slave-rpk1"
server_key: "8de14074c477b3a09904c3b70f0ba15a"
server_key_iv: "ceafcfb5aa37320818d24a2045b96cf9"
server_key_version: "encryptionkey@0"
I20260925 00:57:21.972503 4424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:22.015223 4424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:22.016007 4424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:22.016211 4424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:22.023840 4424 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:40031
I20260925 00:57:22.023860 4490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:40031 every 8 connection(s)
I20260925 00:57:22.025012 4424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:22.028180 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4424
I20260925 00:57:22.028301 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:22.028350 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:22.028604 17192 external_mini_cluster.cc:1428] Setting key a7cb6a5eee5d998ab32ee99d25218b70
I20260925 00:57:22.035365 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2: Bootstrap starting.
I20260925 00:57:22.037997 4491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:22.038587 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.030205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.16.202.62:40031)
0925 00:57:22.030869 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.030887 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.030933 (+ 46us) server_negotiation.cc:407] Connection header received
0925 00:57:22.031634 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.031653 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.032080 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.032381 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:22.032901 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.033915 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.035148 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.035679 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.035806 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.036667 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:22.036699 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:22.036717 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:22.037136 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:22.037194 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:22.037215 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:22.037290 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.037459 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.037732 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
I20260925 00:57:22.038781 4491 log.cc:824] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:22.041141 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2: No bootstrap required, opened a new log
I20260925 00:57:22.044849 4491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396c09392c5549a083b808fc4a553da2" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40031 } }
I20260925 00:57:22.045190 4491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:22.045253 4491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396c09392c5549a083b808fc4a553da2, State: Initialized, Role: FOLLOWER
I20260925 00:57:22.045710 4491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396c09392c5549a083b808fc4a553da2" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40031 } }
I20260925 00:57:22.045852 4491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:22.045902 4491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:22.045976 4491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:22.046901 4491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396c09392c5549a083b808fc4a553da2" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40031 } }
I20260925 00:57:22.047219 4491 leader_election.cc:302] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 396c09392c5549a083b808fc4a553da2; no voters:
I20260925 00:57:22.047502 4491 leader_election.cc:288] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:22.047626 4496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:22.047854 4496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [term 1 LEADER]: Becoming Leader. State: Replica: 396c09392c5549a083b808fc4a553da2, State: Running, Role: LEADER
I20260925 00:57:22.048177 4496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396c09392c5549a083b808fc4a553da2" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40031 } }
I20260925 00:57:22.048758 4491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:22.049783 4497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "396c09392c5549a083b808fc4a553da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396c09392c5549a083b808fc4a553da2" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40031 } } }
I20260925 00:57:22.049926 4497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:22.050211 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396c09392c5549a083b808fc4a553da2. Latest consensus state: current_term: 1 leader_uuid: "396c09392c5549a083b808fc4a553da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396c09392c5549a083b808fc4a553da2" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 40031 } } }
I20260925 00:57:22.050326 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:22.050412 4505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:22.053630 4505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:22.059882 4505 catalog_manager.cc:1347] Generated new cluster ID: 606e9a06b3c74df6802ea41ef27ffd4b
I20260925 00:57:22.059967 4505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:22.091975 4505 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:22.093159 4505 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:22.097512 4505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2: Generated new TSK 0
I20260925 00:57:22.098441 4505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:22.103947 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.1:0
--local_ip_for_outbound_sockets=127.16.202.1
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40031
--builtin_ntp_servers=127.16.202.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:57:22.217418 4515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:22.217684 4515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:22.217721 4515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:22.221753 4515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:22.221812 4515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:22.221868 4515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.1
I20260925 00:57:22.226619 4515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.202.1
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.1
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:22.227659 4515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:22.229010 4515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:22.238798 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:22.238798 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:22.238855 4515 server_base.cc:1034] running on GCE node
W20260925 00:57:22.238798 4523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:22.239511 4515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:22.240170 4515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:22.241325 4515 hybrid_clock.cc:648] HybridClock initialized: now 1790297842241312 us; error 27 us; skew 500 ppm
I20260925 00:57:22.243296 4515 webserver.cc:460] Webserver started at http://127.16.202.1:33573/ using document root <none> and password file <none>
I20260925 00:57:22.243885 4515 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:22.243953 4515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:22.244179 4515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:22.246320 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data/instance:
uuid: "6d3429f139e8449cbf38c0340cd843b8"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "92d79fe1e218569e4ed29e7d70ec4f16"
server_key_iv: "a9be964e7e6bb84aedd4902d1d211670"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.246853 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance:
uuid: "6d3429f139e8449cbf38c0340cd843b8"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "92d79fe1e218569e4ed29e7d70ec4f16"
server_key_iv: "a9be964e7e6bb84aedd4902d1d211670"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.250821 4515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:22.253309 4529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.254568 4515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:22.254676 4515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal
uuid: "6d3429f139e8449cbf38c0340cd843b8"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "92d79fe1e218569e4ed29e7d70ec4f16"
server_key_iv: "a9be964e7e6bb84aedd4902d1d211670"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.254778 4515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:22.281893 4515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:22.282649 4515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:22.282830 4515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:22.283433 4515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:22.284420 4515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:22.284468 4515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.284586 4515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:22.284616 4515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.294229 4515 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.1:43485
I20260925 00:57:22.294255 4642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.1:43485 every 8 connection(s)
I20260925 00:57:22.295331 4515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/data/info.pb
I20260925 00:57:22.301179 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4515
I20260925 00:57:22.301295 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-0/wal/instance
I20260925 00:57:22.301672 17192 external_mini_cluster.cc:1428] Setting key b8fdb5cbc8327cb464f8b4575ac6653c
I20260925 00:57:22.303758 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.2:0
--local_ip_for_outbound_sockets=127.16.202.2
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40031
--builtin_ntp_servers=127.16.202.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:57:22.305648 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.297322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.1:47771 (local address 127.16.202.62:40031)
0925 00:57:22.297468 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.297473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.298672 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 00:57:22.300164 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.300167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.300217 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.300299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:22.301459 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.302188 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.303601 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.303824 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.303913 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.305152 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:22.305155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:22.305157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:22.305171 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:22.305191 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:22.305199 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:22.305259 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.305415 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.305492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:57:22.306797 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.297787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40031 (local address 127.16.202.1:47771)
0925 00:57:22.298508 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.298555 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.299918 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.300474 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.300482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.300575 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:22.301222 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.301238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.302362 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.302366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.303428 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.303439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.303978 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.303982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.304155 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.304846 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:22.304847 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:22.304896 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:22.304975 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:22.305270 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:22.305280 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:22.305286 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.305404 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.305771 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260925 00:57:22.308250 4643 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40031
I20260925 00:57:22.308512 4643 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:22.309127 4643 heartbeater.cc:511] Master 127.16.202.62:40031 requested a full tablet report, sending...
I20260925 00:57:22.310768 4452 ts_manager.cc:194] Registered new tserver with Master: 6d3429f139e8449cbf38c0340cd843b8 (127.16.202.1:43485)
I20260925 00:57:22.311939 4452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.1:47771
W20260925 00:57:22.417012 4646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:22.417299 4646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:22.417337 4646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:22.421464 4646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:22.421540 4646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:22.421602 4646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.2
I20260925 00:57:22.426573 4646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.202.2
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.2
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:22.427724 4646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:22.429212 4646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:22.439252 4652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:22.439255 4651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:22.439255 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:22.439465 4646 server_base.cc:1034] running on GCE node
I20260925 00:57:22.440029 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:22.440706 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:22.441882 4646 hybrid_clock.cc:648] HybridClock initialized: now 1790297842441850 us; error 48 us; skew 500 ppm
I20260925 00:57:22.443789 4646 webserver.cc:460] Webserver started at http://127.16.202.2:41551/ using document root <none> and password file <none>
I20260925 00:57:22.444348 4646 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:22.444415 4646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:22.444612 4646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:22.446738 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data/instance:
uuid: "060ca713e8b44013a1c44953ab201e1e"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "ebc5c40cd424f77fc55502892f36af11"
server_key_iv: "fa9f10a58463e6b28e31dda350a7bd3d"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.447309 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance:
uuid: "060ca713e8b44013a1c44953ab201e1e"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "ebc5c40cd424f77fc55502892f36af11"
server_key_iv: "fa9f10a58463e6b28e31dda350a7bd3d"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.451262 4646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:22.453770 4660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.455044 4646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:22.455152 4646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal
uuid: "060ca713e8b44013a1c44953ab201e1e"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "ebc5c40cd424f77fc55502892f36af11"
server_key_iv: "fa9f10a58463e6b28e31dda350a7bd3d"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.455250 4646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:22.471611 4646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:22.472394 4646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:22.472573 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:22.473224 4646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:22.474238 4646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:22.474288 4646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.474340 4646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:22.474368 4646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.483242 4646 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.2:32893
I20260925 00:57:22.483280 4773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.2:32893 every 8 connection(s)
I20260925 00:57:22.484314 4646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/data/info.pb
I20260925 00:57:22.491109 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4646
I20260925 00:57:22.491230 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal/instance
I20260925 00:57:22.491511 17192 external_mini_cluster.cc:1428] Setting key c1efee26fe0edd55ef7f28a3051c853b
I20260925 00:57:22.493793 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.202.3:0
--local_ip_for_outbound_sockets=127.16.202.3
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40031
--builtin_ntp_servers=127.16.202.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:57:22.494354 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.486471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.2:41101 (local address 127.16.202.62:40031)
0925 00:57:22.486631 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.486636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.487683 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 00:57:22.489121 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.489124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.489174 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.489250 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:22.490172 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.490746 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.492055 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.492346 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.492459 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.493816 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:22.493821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:22.493823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:22.493839 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:22.493865 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:22.493876 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:22.493954 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.494170 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.494228 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:57:22.495513 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.486866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40031 (local address 127.16.202.2:41101)
0925 00:57:22.487537 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.487579 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.488898 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.489416 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.489423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.489512 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:22.490020 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.490033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.490912 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.490915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.491908 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.491921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.492526 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.492531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.492759 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.493526 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:22.493529 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:22.493575 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:22.493652 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:22.493999 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:22.494009 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:22.494015 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.494147 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.494464 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260925 00:57:22.496920 4774 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40031
I20260925 00:57:22.497170 4774 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:22.497781 4774 heartbeater.cc:511] Master 127.16.202.62:40031 requested a full tablet report, sending...
I20260925 00:57:22.498991 4452 ts_manager.cc:194] Registered new tserver with Master: 060ca713e8b44013a1c44953ab201e1e (127.16.202.2:32893)
I20260925 00:57:22.499799 4452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.2:41101
W20260925 00:57:22.608877 4777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:22.609185 4777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:22.609227 4777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:22.613400 4777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:22.613476 4777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:22.613538 4777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.202.3
I20260925 00:57:22.618471 4777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.202.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.202.3
--webserver_port=0
--tserver_master_addrs=127.16.202.62:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.202.3
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:22.619545 4777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:22.620949 4777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:22.630612 4783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:22.630659 4777 server_base.cc:1034] running on GCE node
W20260925 00:57:22.630604 4782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:22.630604 4785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:22.631310 4777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:22.631964 4777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:22.633152 4777 hybrid_clock.cc:648] HybridClock initialized: now 1790297842633121 us; error 41 us; skew 500 ppm
I20260925 00:57:22.644039 4777 webserver.cc:460] Webserver started at http://127.16.202.3:35401/ using document root <none> and password file <none>
I20260925 00:57:22.644747 4777 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:22.644834 4777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:22.645071 4777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:22.647311 4777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data/instance:
uuid: "c4ab292e2b2646128bfbba877a114f60"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "d4f14101024cd1a9305ad01cf2716b9a"
server_key_iv: "a523e8531a8efc185e423f4697779cd9"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.647912 4777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance:
uuid: "c4ab292e2b2646128bfbba877a114f60"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "d4f14101024cd1a9305ad01cf2716b9a"
server_key_iv: "a523e8531a8efc185e423f4697779cd9"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.652160 4777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:22.654933 4791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.656447 4777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:57:22.656629 4777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal
uuid: "c4ab292e2b2646128bfbba877a114f60"
format_stamp: "Formatted at 2026-09-25 00:57:22 on dist-test-slave-rpk1"
server_key: "d4f14101024cd1a9305ad01cf2716b9a"
server_key_iv: "a523e8531a8efc185e423f4697779cd9"
server_key_version: "encryptionkey@0"
I20260925 00:57:22.656826 4777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:22.674153 4777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:22.674878 4777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:22.675055 4777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:22.675710 4777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:57:22.676826 4777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:57:22.676883 4777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.676930 4777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:57:22.676959 4777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:22.685447 4777 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.3:34189
I20260925 00:57:22.685554 4904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.3:34189 every 8 connection(s)
I20260925 00:57:22.686425 4777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/data/info.pb
I20260925 00:57:22.691294 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4777
I20260925 00:57:22.691404 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal/instance
I20260925 00:57:22.691740 17192 external_mini_cluster.cc:1428] Setting key fedb6b2b2866fb831a70fa36d85b41b0
I20260925 00:57:22.696508 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.688362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:34687 (local address 127.16.202.62:40031)
0925 00:57:22.688494 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.688497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.689797 (+ 1300us) server_negotiation.cc:407] Connection header received
0925 00:57:22.691306 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.691309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.691374 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.691448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:22.692491 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.693149 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.694396 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.694621 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.694711 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.695995 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:22.695999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:22.696001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:22.696014 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:22.696035 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:22.696043 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:22.696104 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.696263 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.696335 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:57:22.697579 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.688870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:40031 (local address 127.16.202.3:34687)
0925 00:57:22.689620 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.689666 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.691056 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.691667 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.691676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.691774 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:22.692329 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.692344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.693309 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.693312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.694265 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.694275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.694777 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.694782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.694963 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.695692 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:57:22.695694 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:22.695744 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:57:22.695823 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:22.696115 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:22.696126 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:22.696132 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.696253 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.696589 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260925 00:57:22.698873 4905 heartbeater.cc:347] Connected to a master server at 127.16.202.62:40031
I20260925 00:57:22.699126 4905 heartbeater.cc:464] Registering TS with master...
I20260925 00:57:22.699672 4905 heartbeater.cc:511] Master 127.16.202.62:40031 requested a full tablet report, sending...
I20260925 00:57:22.700850 4452 ts_manager.cc:194] Registered new tserver with Master: c4ab292e2b2646128bfbba877a114f60 (127.16.202.3:34189)
I20260925 00:57:22.701488 4452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.202.3:34687
I20260925 00:57:22.705637 17192 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:57:22.711464 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.707681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.16.202.62:40031)
0925 00:57:22.707822 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.707827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.708002 (+ 175us) server_negotiation.cc:407] Connection header received
0925 00:57:22.708134 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.708136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.708190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.708258 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:22.708540 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.709103 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.709967 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.710178 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.710264 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.711056 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:22.711059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:22.711061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:22.711074 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:22.711092 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:22.711099 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:22.711157 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.711253 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.711311 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:57:22.714607 4452 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:57:22.717664 4452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:57:22.732818 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.726807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32893 (local address 127.0.0.1:39372)
0925 00:57:22.727468 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.727522 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.727647 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.728613 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.728616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.728632 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:22.728755 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.728762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.730186 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.730190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.731298 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.731305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.732385 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.732389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.732475 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.732506 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.732570 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.732640 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":35,"thread_start_us":117,"threads_started":1}
I20260925 00:57:22.732853 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.727633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.3:34189 (local address 127.0.0.1:44116)
0925 00:57:22.727924 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.727936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.728013 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.728930 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.728933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.728950 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:22.729036 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.729042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.730336 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.730339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.731464 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.731475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.732472 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.732475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.732562 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.732583 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.732643 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.732732 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":21,"thread_start_us":88,"threads_started":1}
I20260925 00:57:22.732954 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.728111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.16.202.3:34189)
0925 00:57:22.728459 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.728471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.728490 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:57:22.728557 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.728562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.728768 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.728943 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:22.729158 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.730237 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.731597 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.732328 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.732457 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.732635 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.732714 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.732810 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":156,"threads_started":1}
I20260925 00:57:22.733327 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.726958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.16.202.1:43485)
0925 00:57:22.727935 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.727946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.727987 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:57:22.728067 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.728072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.728287 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.728439 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:22.728806 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.729684 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.731231 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.731980 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.732091 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.732300 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.732758 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.732860 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":101,"threads_started":1}
I20260925 00:57:22.733464 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.727084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.16.202.2:32893)
0925 00:57:22.728123 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.728133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.728152 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:57:22.728217 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.728221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.728377 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.728506 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:22.728866 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.730064 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.731443 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.732096 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.732199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.732399 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.733203 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.733313 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":133,"threads_started":1}
I20260925 00:57:22.733534 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.726807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43485 (local address 127.0.0.1:42970)
0925 00:57:22.727578 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.727593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.727682 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.728485 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.728491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.728517 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:22.728667 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.728700 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.729794 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.729797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.731093 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.731104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.732109 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.732112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.732231 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.732270 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.732375 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.732561 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":102,"thread_start_us":98,"threads_started":1}
I20260925 00:57:22.736166 4839 tablet_service.cc:1467] Processing CreateTablet for tablet a1f06170ae2b4a18bf08fe20767f2b40 (DEFAULT_TABLE table=test-table [id=51e2a24739e34d0a86692832f380de0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:22.736212 4708 tablet_service.cc:1467] Processing CreateTablet for tablet a1f06170ae2b4a18bf08fe20767f2b40 (DEFAULT_TABLE table=test-table [id=51e2a24739e34d0a86692832f380de0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:22.736212 4577 tablet_service.cc:1467] Processing CreateTablet for tablet a1f06170ae2b4a18bf08fe20767f2b40 (DEFAULT_TABLE table=test-table [id=51e2a24739e34d0a86692832f380de0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:57:22.737200 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f06170ae2b4a18bf08fe20767f2b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:22.737239 4577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f06170ae2b4a18bf08fe20767f2b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:22.737239 4708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f06170ae2b4a18bf08fe20767f2b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:22.744345 4922 tablet_bootstrap.cc:492] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Bootstrap starting.
I20260925 00:57:22.745502 4924 tablet_bootstrap.cc:492] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: Bootstrap starting.
I20260925 00:57:22.745502 4923 tablet_bootstrap.cc:492] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Bootstrap starting.
I20260925 00:57:22.746624 4922 tablet_bootstrap.cc:654] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:22.747392 4922 log.cc:824] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:22.747756 4924 tablet_bootstrap.cc:654] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:22.747756 4923 tablet_bootstrap.cc:654] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:22.748651 4923 log.cc:824] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:22.748651 4924 log.cc:824] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:22.749558 4922 tablet_bootstrap.cc:492] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: No bootstrap required, opened a new log
I20260925 00:57:22.749781 4922 ts_tablet_manager.cc:1397] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:57:22.751010 4924 tablet_bootstrap.cc:492] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: No bootstrap required, opened a new log
I20260925 00:57:22.751022 4923 tablet_bootstrap.cc:492] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: No bootstrap required, opened a new log
I20260925 00:57:22.751201 4923 ts_tablet_manager.cc:1397] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 00:57:22.751204 4924 ts_tablet_manager.cc:1397] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 00:57:22.754367 4923 raft_consensus.cc:348] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.754274 4922 raft_consensus.cc:348] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.754582 4923 raft_consensus.cc:374] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:22.754583 4922 raft_consensus.cc:374] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:22.754635 4923 raft_consensus.cc:729] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060ca713e8b44013a1c44953ab201e1e, State: Initialized, Role: FOLLOWER
I20260925 00:57:22.754635 4922 raft_consensus.cc:729] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ab292e2b2646128bfbba877a114f60, State: Initialized, Role: FOLLOWER
I20260925 00:57:22.755103 4923 consensus_queue.cc:260] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.755115 4922 consensus_queue.cc:260] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.755226 4924 raft_consensus.cc:348] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.755470 4924 raft_consensus.cc:374] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:22.755529 4924 raft_consensus.cc:729] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3429f139e8449cbf38c0340cd843b8, State: Initialized, Role: FOLLOWER
I20260925 00:57:22.755954 4774 heartbeater.cc:503] Master 127.16.202.62:40031 was elected leader, sending a full tablet report...
I20260925 00:57:22.755962 4924 consensus_queue.cc:260] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.756175 4922 ts_tablet_manager.cc:1428] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 00:57:22.756714 4905 heartbeater.cc:503] Master 127.16.202.62:40031 was elected leader, sending a full tablet report...
I20260925 00:57:22.756852 4924 ts_tablet_manager.cc:1428] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 00:57:22.757161 4643 heartbeater.cc:503] Master 127.16.202.62:40031 was elected leader, sending a full tablet report...
I20260925 00:57:22.758425 4923 ts_tablet_manager.cc:1428] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260925 00:57:22.789511 4929 raft_consensus.cc:482] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:57:22.789714 4929 raft_consensus.cc:504] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.790843 4929 leader_election.cc:288] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d3429f139e8449cbf38c0340cd843b8 (127.16.202.1:43485), 060ca713e8b44013a1c44953ab201e1e (127.16.202.2:32893)
I20260925 00:57:22.794296 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.791184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:58749 (local address 127.16.202.1:43485)
0925 00:57:22.791308 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.791312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.791327 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:57:22.791386 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.791389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.791443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.791526 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:22.791799 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.792300 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.793382 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.793781 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.793896 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.793947 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.794108 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.794184 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:22.794296 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.791080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43485 (local address 127.16.202.3:58749)
0925 00:57:22.791196 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.791211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.791302 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.791552 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.791555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.791575 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:22.791682 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.791688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.792407 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.792411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.793252 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.793261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.793892 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.793895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.793993 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.794011 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.794075 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.794154 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 00:57:22.794661 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.791219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.202.3:40915 (local address 127.16.202.2:32893)
0925 00:57:22.791400 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.791404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.791772 (+ 368us) server_negotiation.cc:407] Connection header received
0925 00:57:22.791886 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.791889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.791928 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.791988 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:57:22.792231 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.792747 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.793722 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.794111 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.794216 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.794265 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.794473 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.794548 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:57:22.795015 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.791092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.2:32893 (local address 127.16.202.3:40915)
0925 00:57:22.791686 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.791697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.791784 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.792013 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:22.792016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:22.792030 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:57:22.792136 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.792141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.792859 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.792863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.793610 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:22.793619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.794229 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.794232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:22.794324 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.794341 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:57:22.794813 (+ 472us) client_negotiation.cc:240] Negotiation successful
0925 00:57:22.794888 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":273,"thread_start_us":105,"threads_started":1}
I20260925 00:57:22.794991 4597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f06170ae2b4a18bf08fe20767f2b40" candidate_uuid: "c4ab292e2b2646128bfbba877a114f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3429f139e8449cbf38c0340cd843b8" is_pre_election: true
I20260925 00:57:22.795300 4597 raft_consensus.cc:2454] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4ab292e2b2646128bfbba877a114f60 in term 0.
I20260925 00:57:22.795615 4728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f06170ae2b4a18bf08fe20767f2b40" candidate_uuid: "c4ab292e2b2646128bfbba877a114f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ca713e8b44013a1c44953ab201e1e" is_pre_election: true
I20260925 00:57:22.795809 4793 leader_election.cc:302] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3429f139e8449cbf38c0340cd843b8, c4ab292e2b2646128bfbba877a114f60; no voters:
I20260925 00:57:22.795872 4728 raft_consensus.cc:2454] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4ab292e2b2646128bfbba877a114f60 in term 0.
I20260925 00:57:22.796072 4929 raft_consensus.cc:2781] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:57:22.796167 4929 raft_consensus.cc:482] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:57:22.796221 4929 raft_consensus.cc:3037] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 0 FOLLOWER]: Advancing to term 1
W20260925 00:57:22.796481 4644 tablet.cc:2369] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:22.797305 4929 raft_consensus.cc:504] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.797645 4929 leader_election.cc:288] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [CANDIDATE]: Term 1 election: Requested vote from peers 6d3429f139e8449cbf38c0340cd843b8 (127.16.202.1:43485), 060ca713e8b44013a1c44953ab201e1e (127.16.202.2:32893)
I20260925 00:57:22.798004 4728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f06170ae2b4a18bf08fe20767f2b40" candidate_uuid: "c4ab292e2b2646128bfbba877a114f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ca713e8b44013a1c44953ab201e1e"
I20260925 00:57:22.798077 4597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f06170ae2b4a18bf08fe20767f2b40" candidate_uuid: "c4ab292e2b2646128bfbba877a114f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3429f139e8449cbf38c0340cd843b8"
I20260925 00:57:22.798132 4728 raft_consensus.cc:3037] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:22.798193 4597 raft_consensus.cc:3037] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:22.799394 4728 raft_consensus.cc:2454] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ab292e2b2646128bfbba877a114f60 in term 1.
I20260925 00:57:22.799394 4597 raft_consensus.cc:2454] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ab292e2b2646128bfbba877a114f60 in term 1.
I20260925 00:57:22.799700 4795 leader_election.cc:302] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ca713e8b44013a1c44953ab201e1e, c4ab292e2b2646128bfbba877a114f60; no voters:
I20260925 00:57:22.799841 4929 raft_consensus.cc:2781] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:22.800079 4929 raft_consensus.cc:686] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 1 LEADER]: Becoming Leader. State: Replica: c4ab292e2b2646128bfbba877a114f60, State: Running, Role: LEADER
I20260925 00:57:22.800415 4929 consensus_queue.cc:237] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } }
I20260925 00:57:22.803674 4452 catalog_manager.cc:5495] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4ab292e2b2646128bfbba877a114f60 (127.16.202.3). New cstate: current_term: 1 leader_uuid: "c4ab292e2b2646128bfbba877a114f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4ab292e2b2646128bfbba877a114f60" member_type: VOTER last_known_addr { host: "127.16.202.3" port: 34189 } health_report { overall_health: HEALTHY } } }
I20260925 00:57:22.834200 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:22.830330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.16.202.3:34189)
0925 00:57:22.830482 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:22.830486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:22.830501 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:57:22.830581 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:22.830583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:22.830641 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:22.830744 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:22.831212 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.831728 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.832620 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:22.832874 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:22.832959 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:22.833261 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:22.833708 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:22.833794 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:22.833925 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 00:57:22.834020 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:57:22.840427 4597 raft_consensus.cc:1261] T a1f06170ae2b4a18bf08fe20767f2b40 P 6d3429f139e8449cbf38c0340cd843b8 [term 1 FOLLOWER]: Refusing update from remote peer c4ab292e2b2646128bfbba877a114f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:22.840426 4728 raft_consensus.cc:1261] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 1 FOLLOWER]: Refusing update from remote peer c4ab292e2b2646128bfbba877a114f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:22.841176 4929 consensus_queue.cc:1035] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3429f139e8449cbf38c0340cd843b8" member_type: VOTER last_known_addr { host: "127.16.202.1" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:22.841377 4932 consensus_queue.cc:1035] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060ca713e8b44013a1c44953ab201e1e" member_type: VOTER last_known_addr { host: "127.16.202.2" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:22.848809 4935 mvcc.cc:205] Tried to move back new op lower bound from 7333059964266672128 to 7333059964113227776. Current Snapshot: MvccSnapshot[applied={T|T < 7333059964266672128}]
I20260925 00:57:22.849380 4936 mvcc.cc:205] Tried to move back new op lower bound from 7333059964266672128 to 7333059964113227776. Current Snapshot: MvccSnapshot[applied={T|T < 7333059964266672128}]
I20260925 00:57:22.850412 4937 mvcc.cc:205] Tried to move back new op lower bound from 7333059964266672128 to 7333059964113227776. Current Snapshot: MvccSnapshot[applied={T|T < 7333059964266672128}]
I20260925 00:57:22.858923 4455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:22.859112 4455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:57:22.861769 4455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 396c09392c5549a083b808fc4a553da2: Sending DeleteTablet for 3 replicas of tablet a1f06170ae2b4a18bf08fe20767f2b40
I20260925 00:57:22.862651 4839 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f06170ae2b4a18bf08fe20767f2b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:22 UTC) from {username='slave'} at 127.0.0.1:44116
I20260925 00:57:22.862653 4577 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f06170ae2b4a18bf08fe20767f2b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:22 UTC) from {username='slave'} at 127.0.0.1:42970
I20260925 00:57:22.862694 4708 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f06170ae2b4a18bf08fe20767f2b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:57:22 UTC) from {username='slave'} at 127.0.0.1:39372
I20260925 00:57:22.863163 4944 tablet_replica.cc:330] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: stopping tablet replica
I20260925 00:57:22.863471 4944 raft_consensus.cc:2229] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:57:22.863767 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4515
I20260925 00:57:22.863816 4944 raft_consensus.cc:2258] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:57:22.866796 4944 ts_tablet_manager.cc:1905] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:22.869211 4944 ts_tablet_manager.cc:1918] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:22.869320 4944 log.cc:1196] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-1/wal/wals/a1f06170ae2b4a18bf08fe20767f2b40
I20260925 00:57:22.869585 4944 ts_tablet_manager.cc:1939] T a1f06170ae2b4a18bf08fe20767f2b40 P 060ca713e8b44013a1c44953ab201e1e: Deleting consensus metadata
W20260925 00:57:22.870910 4440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.202.1:43485 (error 108)
I20260925 00:57:22.870875 4442 catalog_manager.cc:4841] TS 060ca713e8b44013a1c44953ab201e1e (127.16.202.2:32893): tablet a1f06170ae2b4a18bf08fe20767f2b40 (table test-table [id=51e2a24739e34d0a86692832f380de0e]) successfully deleted
I20260925 00:57:22.871695 4945 tablet_replica.cc:330] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: stopping tablet replica
I20260925 00:57:22.872090 4945 raft_consensus.cc:2229] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:57:22.872570 4945 raft_consensus.cc:2258] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 00:57:22.873011 4916 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:22.871735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.1:43485 (local address 127.0.0.1:42986)
0925 00:57:22.871869 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:22.871882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:22.871959 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:22.872899 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.1:43485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:57:22.873204 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4646
W20260925 00:57:22.873281 4440 catalog_manager.cc:4584] TS 6d3429f139e8449cbf38c0340cd843b8 (127.16.202.1:43485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1f06170ae2b4a18bf08fe20767f2b40: Network error: Client connection negotiation failed: client connection to 127.16.202.1:43485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:57:22.875662 4945 ts_tablet_manager.cc:1905] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:57:22.878744 4945 ts_tablet_manager.cc:1918] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:57:22.878880 4945 log.cc:1196] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Deleting WAL directory at /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297591180183-17192-0/minicluster-data/ts-2/wal/wals/a1f06170ae2b4a18bf08fe20767f2b40
I20260925 00:57:22.879213 4945 ts_tablet_manager.cc:1939] T a1f06170ae2b4a18bf08fe20767f2b40 P c4ab292e2b2646128bfbba877a114f60: Deleting consensus metadata
I20260925 00:57:22.880141 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4777
I20260925 00:57:22.886606 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4424
2026-09-25T00:57:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1092 ms)
[----------] 6 tests from EncryptionPolicyTest (10377 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:22 dist-test-slave-rpk1 krb5kdc[4950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:22 dist-test-slave-rpk1 krb5kdc[4950](info): set up 2 sockets
Sep 25 00:57:22 dist-test-slave-rpk1 krb5kdc[4950](info): commencing operation
krb5kdc: starting...
W20260925 00:57:23.852766 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:23 dist-test-slave-rpk1 krb5kdc[4950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:23.972711 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:24.086953 4963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:24.087265 4963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:24.087325 4963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:24.091781 4963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:24.091884 4963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:24.091905 4963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:24.091924 4963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:24.091939 4963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:24.097801 4963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:24.099009 4963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:24.100062 4963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:24.109167 4971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:24.109213 4969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:24.109213 4968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:24.109645 4963 server_base.cc:1034] running on GCE node
I20260925 00:57:24.110270 4963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:24.111508 4963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:24.112723 4963 hybrid_clock.cc:648] HybridClock initialized: now 1790297844112687 us; error 46 us; skew 500 ppm
Sep 25 00:57:24 dist-test-slave-rpk1 krb5kdc[4950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:24.115808 4963 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:24.116813 4963 webserver.cc:460] Webserver started at http://127.0.0.1:37589/ using document root <none> and password file <none>
I20260925 00:57:24.117386 4963 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:24.117453 4963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:24.117666 4963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:24.119904 4963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "3982b33d5a0a46be9ecedf6b9dfe889a"
format_stamp: "Formatted at 2026-09-25 00:57:24 on dist-test-slave-rpk1"
server_key: "45dafa6d2f191368a83ae5d65cdf7e7b"
server_key_iv: "6a69a39ff4e669130b0e25fd4f5c0c3d"
server_key_version: "encryptionkey@0"
I20260925 00:57:24.120532 4963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "3982b33d5a0a46be9ecedf6b9dfe889a"
format_stamp: "Formatted at 2026-09-25 00:57:24 on dist-test-slave-rpk1"
server_key: "45dafa6d2f191368a83ae5d65cdf7e7b"
server_key_iv: "6a69a39ff4e669130b0e25fd4f5c0c3d"
server_key_version: "encryptionkey@0"
I20260925 00:57:24.124773 4963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 00:57:24.127434 4978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:24.128863 4963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:24.128998 4963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "3982b33d5a0a46be9ecedf6b9dfe889a"
format_stamp: "Formatted at 2026-09-25 00:57:24 on dist-test-slave-rpk1"
server_key: "45dafa6d2f191368a83ae5d65cdf7e7b"
server_key_iv: "6a69a39ff4e669130b0e25fd4f5c0c3d"
server_key_version: "encryptionkey@0"
I20260925 00:57:24.129118 4963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:24.147743 4963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:24.151304 4963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:24.151503 4963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:24.159559 4963 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45061
I20260925 00:57:24.159680 5030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45061 every 8 connection(s)
I20260925 00:57:24.160756 4963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:24.163856 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:24.169734 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 4963
I20260925 00:57:24.169749 5031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a: Bootstrap starting.
I20260925 00:57:24.169866 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:24.170125 17192 external_mini_cluster.cc:1428] Setting key 6ff0d047053339428210cffc76f55451
I20260925 00:57:24.172415 5031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:24.173200 5031 log.cc:824] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:24.175045 5031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a: No bootstrap required, opened a new log
I20260925 00:57:24.178692 5031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 00:57:24.178939 5031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:24.178988 5031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3982b33d5a0a46be9ecedf6b9dfe889a, State: Initialized, Role: FOLLOWER
Sep 25 00:57:24 dist-test-slave-rpk1 krb5kdc[4950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:57:24.179508 5031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 00:57:24.179667 5031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:24.179731 5031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:24.179834 5031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:24.180858 5031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 00:57:24.181249 5031 leader_election.cc:302] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3982b33d5a0a46be9ecedf6b9dfe889a; no voters:
I20260925 00:57:24.181658 5031 leader_election.cc:288] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:24.181772 5036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:24.181948 5036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [term 1 LEADER]: Becoming Leader. State: Replica: 3982b33d5a0a46be9ecedf6b9dfe889a, State: Running, Role: LEADER
I20260925 00:57:24.182276 5036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 00:57:24.183081 5031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:24.184139 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3982b33d5a0a46be9ecedf6b9dfe889a. Latest consensus state: current_term: 1 leader_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } } }
I20260925 00:57:24.184266 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:24.184702 5045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:24.185616 5037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3982b33d5a0a46be9ecedf6b9dfe889a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } } }
I20260925 00:57:24.185720 5037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:24.186478 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:24.171598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40249 (local address 127.0.0.1:45061)
0925 00:57:24.172123 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:24.172140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:24.172181 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:57:24.173012 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:24.173034 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:24.173450 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:24.173785 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:24.174681 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:24.175789 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:24.176729 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:24.177241 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:24.177417 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:24.180257 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:24.180262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:24.180277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:24.180305 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:24.182419 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:24.183055 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:24.183061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:24.183068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:24.183242 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:24.183655 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:24.183658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:24.183661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:24.184247 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:24.184454 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:24.185364 (+ 910us) server_negotiation.cc:299] Negotiation successful
0925 00:57:24.185564 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":153,"threads_started":1}
I20260925 00:57:24.187932 5045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:24.190076 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:24.193862 5045 catalog_manager.cc:1347] Generated new cluster ID: e194a1a15b8d4a4b9dff2ab41d343169
I20260925 00:57:24.193931 5045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:24.201354 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:24.192435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41435 (local address 127.0.0.1:45061)
0925 00:57:24.192551 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:24.192555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:24.192780 (+ 225us) server_negotiation.cc:407] Connection header received
0925 00:57:24.192925 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:24.192928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:24.192973 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:24.193035 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:24.193707 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:24.194437 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:24.195703 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:24.196047 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:24.196110 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:24.197876 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:24.197880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:24.197883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:24.197907 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:24.199443 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:24.199943 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:24.199946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:24.199948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:24.200033 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:24.200343 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:24.200347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:24.200348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:24.200535 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:24.200651 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:24.201090 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 00:57:24.201190 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:57:24.207614 5045 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:24.208782 5045 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:24.220855 5045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3982b33d5a0a46be9ecedf6b9dfe889a: Generated new TSK 0
I20260925 00:57:24.221721 5045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:24.235307 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 4963
W20260925 00:57:24.240345 4948 connection.cc:535] client connection to 127.0.0.1:45061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45061): Connection reset by peer (error 104)
2026-09-25T00:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:24 dist-test-slave-rpk1 krb5kdc[5062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:24 dist-test-slave-rpk1 krb5kdc[5062](info): set up 2 sockets
Sep 25 00:57:24 dist-test-slave-rpk1 krb5kdc[5062](info): commencing operation
krb5kdc: starting...
W20260925 00:57:25.196743 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:25 dist-test-slave-rpk1 krb5kdc[5062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:25.319126 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:25.429217 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:25.429558 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:25.429622 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:25.433771 5075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:25.433861 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:25.433890 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:25.433914 5075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:25.433943 5075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:25.439540 5075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:25.440719 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:25.441805 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:25.450488 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:25.450536 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:25.450425 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:25.450716 5075 server_base.cc:1034] running on GCE node
I20260925 00:57:25.451341 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:25.452486 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:25.453647 5075 hybrid_clock.cc:648] HybridClock initialized: now 1790297845453625 us; error 45 us; skew 500 ppm
Sep 25 00:57:25 dist-test-slave-rpk1 krb5kdc[5062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:25.456424 5075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:25.457403 5075 webserver.cc:460] Webserver started at http://127.0.0.1:38137/ using document root <none> and password file <none>
I20260925 00:57:25.457981 5075 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:25.458053 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:25.458262 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:25.460433 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "167ac1bb6cf64ebabbd626004ffacb58"
format_stamp: "Formatted at 2026-09-25 00:57:25 on dist-test-slave-rpk1"
server_key: "b043192cd574898e5214c2aab34662ce"
server_key_iv: "59e0b0cc503b59a26beb6d711eaa7e87"
server_key_version: "encryptionkey@0"
I20260925 00:57:25.461062 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "167ac1bb6cf64ebabbd626004ffacb58"
format_stamp: "Formatted at 2026-09-25 00:57:25 on dist-test-slave-rpk1"
server_key: "b043192cd574898e5214c2aab34662ce"
server_key_iv: "59e0b0cc503b59a26beb6d711eaa7e87"
server_key_version: "encryptionkey@0"
I20260925 00:57:25.465011 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:25.467692 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:25.469045 5075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:25.469175 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "167ac1bb6cf64ebabbd626004ffacb58"
format_stamp: "Formatted at 2026-09-25 00:57:25 on dist-test-slave-rpk1"
server_key: "b043192cd574898e5214c2aab34662ce"
server_key_iv: "59e0b0cc503b59a26beb6d711eaa7e87"
server_key_version: "encryptionkey@0"
I20260925 00:57:25.469288 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:25.484774 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:25.488021 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:25.488207 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:25.495234 5075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38653
I20260925 00:57:25.495249 5142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38653 every 8 connection(s)
I20260925 00:57:25.496438 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:25.499171 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:25.504648 5143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58: Bootstrap starting.
I20260925 00:57:25.505913 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5075
I20260925 00:57:25.506044 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:25.506295 17192 external_mini_cluster.cc:1428] Setting key 9a693306ff5ea3a4783ee880996c48e4
I20260925 00:57:25.507416 5143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:25.508163 5143 log.cc:824] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:25.510242 5143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58: No bootstrap required, opened a new log
I20260925 00:57:25.514426 5143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167ac1bb6cf64ebabbd626004ffacb58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 00:57:25.514729 5143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:25.514789 5143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167ac1bb6cf64ebabbd626004ffacb58, State: Initialized, Role: FOLLOWER
I20260925 00:57:25.515251 5143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167ac1bb6cf64ebabbd626004ffacb58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 00:57:25.515383 5143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:25.515451 5143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:25.515489 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:25.507418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35217 (local address 127.0.0.1:38653)
0925 00:57:25.507890 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:25.507905 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:25.507942 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:57:25.508668 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:25.508715 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:25.509090 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:25.509381 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:25.509726 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:25.510715 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:25.511911 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:25.512417 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:25.512553 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:25.513558 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:25.513563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:25.513578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:25.514026 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:25.514087 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:25.514112 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:25.514223 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:25.514419 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 00:57:25.514579 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":137,"threads_started":1}
I20260925 00:57:25.515556 5143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:25.516767 5143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167ac1bb6cf64ebabbd626004ffacb58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 00:57:25.517148 5143 leader_election.cc:302] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 167ac1bb6cf64ebabbd626004ffacb58; no voters:
I20260925 00:57:25.517462 5143 leader_election.cc:288] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:25.517561 5148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:25.517761 5148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [term 1 LEADER]: Becoming Leader. State: Replica: 167ac1bb6cf64ebabbd626004ffacb58, State: Running, Role: LEADER
I20260925 00:57:25.518142 5148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167ac1bb6cf64ebabbd626004ffacb58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 00:57:25.518813 5143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:25.519025 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:25.519920 5149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "167ac1bb6cf64ebabbd626004ffacb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167ac1bb6cf64ebabbd626004ffacb58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } } }
I20260925 00:57:25.520067 5149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:25.520340 5150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 167ac1bb6cf64ebabbd626004ffacb58. Latest consensus state: current_term: 1 leader_uuid: "167ac1bb6cf64ebabbd626004ffacb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167ac1bb6cf64ebabbd626004ffacb58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } } }
I20260925 00:57:25.520493 5150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58 [sys.catalog]: This master's current role is: LEADER
W20260925 00:57:25.524034 5167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:25.524115 5167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:25.524508 5157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:25.525251 5157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:25.526470 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:25.521596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53443 (local address 127.0.0.1:38653)
0925 00:57:25.521712 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:25.521716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:25.522002 (+ 286us) server_negotiation.cc:407] Connection header received
0925 00:57:25.522122 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:25.522124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:25.522163 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:25.522226 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:25.522500 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:25.523240 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:25.524341 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:25.524698 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:25.524802 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:25.526003 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:25.526007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:25.526010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:25.526023 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:25.526051 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:25.526063 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:25.526149 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:25.526309 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 00:57:25.526347 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:57:25.530820 5157 catalog_manager.cc:1347] Generated new cluster ID: 38dd101fed644dfb87fe659723ea393a
I20260925 00:57:25.530874 5157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:25.556770 5157 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:25.557734 5157 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:25.563993 5157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 167ac1bb6cf64ebabbd626004ffacb58: Generated new TSK 0
I20260925 00:57:25.564880 5157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:25.574328 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5075
W20260925 00:57:25.578960 5060 connection.cc:535] client connection to 127.0.0.1:38653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38653): Connection reset by peer (error 104)
2026-09-25T00:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:25 dist-test-slave-rpk1 krb5kdc[5174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:25 dist-test-slave-rpk1 krb5kdc[5174](info): set up 2 sockets
Sep 25 00:57:25 dist-test-slave-rpk1 krb5kdc[5174](info): commencing operation
krb5kdc: starting...
W20260925 00:57:26.528611 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:26 dist-test-slave-rpk1 krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:26.649585 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:26.763499 5187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:26.763814 5187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:26.763859 5187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:26.768267 5187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:26.768360 5187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:26.768378 5187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:26.768394 5187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:26.768409 5187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:26.774164 5187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:26.775375 5187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:26.776412 5187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:26.785060 5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:26.785071 5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:26.785063 5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:26.785439 5187 server_base.cc:1034] running on GCE node
I20260925 00:57:26.785995 5187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:26.787117 5187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:26.788280 5187 hybrid_clock.cc:648] HybridClock initialized: now 1790297846788263 us; error 33 us; skew 500 ppm
Sep 25 00:57:26 dist-test-slave-rpk1 krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:26.791006 5187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:26.791926 5187 webserver.cc:460] Webserver started at http://127.0.0.1:35201/ using document root <none> and password file <none>
I20260925 00:57:26.792470 5187 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:26.792538 5187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:26.792783 5187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:26.794836 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "a8550878174041639593ba2eea22189b"
format_stamp: "Formatted at 2026-09-25 00:57:26 on dist-test-slave-rpk1"
server_key: "e54570348796dd856dcb066c1e6aeeb4"
server_key_iv: "cc2e8e30eeb7854d939bd7bd43cf60cf"
server_key_version: "encryptionkey@0"
I20260925 00:57:26.795368 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "a8550878174041639593ba2eea22189b"
format_stamp: "Formatted at 2026-09-25 00:57:26 on dist-test-slave-rpk1"
server_key: "e54570348796dd856dcb066c1e6aeeb4"
server_key_iv: "cc2e8e30eeb7854d939bd7bd43cf60cf"
server_key_version: "encryptionkey@0"
I20260925 00:57:26.799206 5187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:26.801743 5202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:26.803118 5187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:26.803257 5187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "a8550878174041639593ba2eea22189b"
format_stamp: "Formatted at 2026-09-25 00:57:26 on dist-test-slave-rpk1"
server_key: "e54570348796dd856dcb066c1e6aeeb4"
server_key_iv: "cc2e8e30eeb7854d939bd7bd43cf60cf"
server_key_version: "encryptionkey@0"
I20260925 00:57:26.803369 5187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:26.808131 5187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:26.811415 5187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:26.811599 5187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:26.818327 5187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39167
I20260925 00:57:26.818341 5254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39167 every 8 connection(s)
I20260925 00:57:26.819291 5187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:26.822211 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:26.826179 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5187
I20260925 00:57:26.826310 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:26.826609 17192 external_mini_cluster.cc:1428] Setting key cf6f5a1eadbcf7af47e12c463440c49e
I20260925 00:57:26.828374 5255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b: Bootstrap starting.
I20260925 00:57:26.830858 5255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:26.831655 5255 log.cc:824] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:26.831756 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:26.827971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60247 (local address 127.0.0.1:39167)
0925 00:57:26.828387 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:26.828402 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:26.828440 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:57:26.829149 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:26.829168 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:26.829467 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:26.829752 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:26.830565 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:26.830570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:26.830582 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:26.830591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:26.830643 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:26.830662 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:26.830745 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:26.830886 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 00:57:26.831052 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260925 00:57:26.833602 5255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b: No bootstrap required, opened a new log
I20260925 00:57:26.834551 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:26.838147 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:26.836520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54219 (local address 127.0.0.1:39167)
0925 00:57:26.836625 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:26.836629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:26.836900 (+ 271us) server_negotiation.cc:407] Connection header received
0925 00:57:26.837020 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:26.837022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:26.837053 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:26.837117 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:26.837780 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:26.837783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:26.837785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:26.837794 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:26.837819 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:26.837830 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:26.837893 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:26.837980 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 00:57:26.838027 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:57:26.837973 5255 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8550878174041639593ba2eea22189b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260925 00:57:26.838289 5255 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:26.838362 5255 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8550878174041639593ba2eea22189b, State: Initialized, Role: FOLLOWER
I20260925 00:57:26.838792 5255 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8550878174041639593ba2eea22189b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260925 00:57:26.838934 5255 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:26.839006 5255 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:26.839115 5255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:26.840055 5255 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8550878174041639593ba2eea22189b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260925 00:57:26.840387 5255 leader_election.cc:302] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8550878174041639593ba2eea22189b; no voters:
I20260925 00:57:26.840642 5255 leader_election.cc:288] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:26.840765 5265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:26.840969 5265 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [term 1 LEADER]: Becoming Leader. State: Replica: a8550878174041639593ba2eea22189b, State: Running, Role: LEADER
I20260925 00:57:26.841321 5265 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8550878174041639593ba2eea22189b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260925 00:57:26.841852 5255 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:26.842996 5267 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8550878174041639593ba2eea22189b. Latest consensus state: current_term: 1 leader_uuid: "a8550878174041639593ba2eea22189b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8550878174041639593ba2eea22189b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } } }
I20260925 00:57:26.843016 5266 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8550878174041639593ba2eea22189b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8550878174041639593ba2eea22189b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } } }
I20260925 00:57:26.843133 5266 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:26.843132 5267 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b [sys.catalog]: This master's current role is: LEADER
W20260925 00:57:26.846853 5280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:26.846938 5280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:26.847026 5274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:26.847754 5274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:26.853833 5274 catalog_manager.cc:1347] Generated new cluster ID: a30301d6f1b244e2b60073193dff1c6a
I20260925 00:57:26.853920 5274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:26.867045 5274 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:26.867381 5274 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:26.877570 5274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8550878174041639593ba2eea22189b: Generated new TSK 0
I20260925 00:57:26.878355 5274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:26.887262 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5187
2026-09-25T00:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:26 dist-test-slave-rpk1 krb5kdc[5286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:26 dist-test-slave-rpk1 krb5kdc[5286](info): set up 2 sockets
Sep 25 00:57:26 dist-test-slave-rpk1 krb5kdc[5286](info): commencing operation
krb5kdc: starting...
W20260925 00:57:27.852999 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:27 dist-test-slave-rpk1 krb5kdc[5286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:27.976969 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:28.090425 5299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:28.090739 5299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:28.090781 5299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:28.094818 5299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:28.094882 5299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:28.094908 5299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:28.094924 5299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:28.094938 5299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:28.100406 5299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:28.101536 5299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:28.102497 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:28.111312 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:28.111333 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:28.111320 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:28.111522 5299 server_base.cc:1034] running on GCE node
I20260925 00:57:28.112298 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:28.113516 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:28.114686 5299 hybrid_clock.cc:648] HybridClock initialized: now 1790297848114667 us; error 33 us; skew 500 ppm
Sep 25 00:57:28 dist-test-slave-rpk1 krb5kdc[5286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:28.122887 5299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:28.123935 5299 webserver.cc:460] Webserver started at http://127.0.0.1:46047/ using document root <none> and password file <none>
I20260925 00:57:28.124516 5299 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:28.124598 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:28.124848 5299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:28.126989 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "1943df213b894a03b4fd5bdd22730d12"
format_stamp: "Formatted at 2026-09-25 00:57:28 on dist-test-slave-rpk1"
server_key: "856db279a867fec143cad69a7b5418c4"
server_key_iv: "d44e81ecc462741cf4dc7722022d035a"
server_key_version: "encryptionkey@0"
I20260925 00:57:28.127528 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "1943df213b894a03b4fd5bdd22730d12"
format_stamp: "Formatted at 2026-09-25 00:57:28 on dist-test-slave-rpk1"
server_key: "856db279a867fec143cad69a7b5418c4"
server_key_iv: "d44e81ecc462741cf4dc7722022d035a"
server_key_version: "encryptionkey@0"
I20260925 00:57:28.131436 5299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:28.134100 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:28.135483 5299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:57:28.135617 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "1943df213b894a03b4fd5bdd22730d12"
format_stamp: "Formatted at 2026-09-25 00:57:28 on dist-test-slave-rpk1"
server_key: "856db279a867fec143cad69a7b5418c4"
server_key_iv: "d44e81ecc462741cf4dc7722022d035a"
server_key_version: "encryptionkey@0"
I20260925 00:57:28.135728 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:28.150065 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:28.153324 5299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:28.153496 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:28.160184 5299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37317
I20260925 00:57:28.160245 5366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37317 every 8 connection(s)
I20260925 00:57:28.161199 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:28.163733 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5299
I20260925 00:57:28.163874 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:28.164161 5367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:28.164167 17192 external_mini_cluster.cc:1428] Setting key af479853824dd4eb69e0fcb0517e32ee
I20260925 00:57:28.171438 5367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12: Bootstrap starting.
Sep 25 00:57:28 dist-test-slave-rpk1 krb5kdc[5286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:57:28.173843 5367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:28.174651 5367 log.cc:824] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:28.176754 5367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12: No bootstrap required, opened a new log
I20260925 00:57:28.178812 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:28.165531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59193 (local address 127.0.0.1:37317)
0925 00:57:28.165949 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:28.165966 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:28.166008 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:57:28.166916 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:28.166941 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:28.167387 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:28.167770 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:28.168474 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:28.169977 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:28.170833 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:28.171520 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:28.171677 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:28.174179 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:28.174184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:28.174199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:28.174231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:28.175876 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:28.176368 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:28.176373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:28.176377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:28.176531 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:28.176833 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:28.176836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:28.176838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:28.177260 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:28.177403 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:28.177711 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 00:57:28.177998 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260925 00:57:28.180801 5367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943df213b894a03b4fd5bdd22730d12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260925 00:57:28.181092 5367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:28.181155 5367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1943df213b894a03b4fd5bdd22730d12, State: Initialized, Role: FOLLOWER
I20260925 00:57:28.181615 5367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943df213b894a03b4fd5bdd22730d12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260925 00:57:28.181777 5367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:28.181972 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:28.182000 5367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:28.182106 5367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:28.183346 5367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943df213b894a03b4fd5bdd22730d12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260925 00:57:28.183768 5367 leader_election.cc:302] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1943df213b894a03b4fd5bdd22730d12; no voters:
I20260925 00:57:28.184154 5367 leader_election.cc:288] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:28.184248 5376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:28.184470 5376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [term 1 LEADER]: Becoming Leader. State: Replica: 1943df213b894a03b4fd5bdd22730d12, State: Running, Role: LEADER
I20260925 00:57:28.184813 5376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943df213b894a03b4fd5bdd22730d12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260925 00:57:28.185578 5367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:28.186437 5379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1943df213b894a03b4fd5bdd22730d12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943df213b894a03b4fd5bdd22730d12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } } }
I20260925 00:57:28.186575 5379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:28.186769 5379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1943df213b894a03b4fd5bdd22730d12. Latest consensus state: current_term: 1 leader_uuid: "1943df213b894a03b4fd5bdd22730d12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943df213b894a03b4fd5bdd22730d12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } } }
I20260925 00:57:28.186838 5379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:28.187211 5386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:28.190898 5386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:28.192888 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:28.184348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48721 (local address 127.0.0.1:37317)
0925 00:57:28.184470 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:28.184474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:28.185122 (+ 648us) server_negotiation.cc:407] Connection header received
0925 00:57:28.185241 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:28.185244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:28.185290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:28.185360 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:28.186155 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:28.186921 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:28.188003 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:28.188311 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:28.188373 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:28.189985 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:28.189989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:28.189991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:28.190017 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:28.191341 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:28.191720 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:28.191724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:28.191726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:28.191809 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:28.192046 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:28.192049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:28.192051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:28.192226 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:28.192339 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:28.192577 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 00:57:28.192725 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:57:28.197317 5386 catalog_manager.cc:1347] Generated new cluster ID: d02fddfedc4a4841b4a260529b2afb14
I20260925 00:57:28.197402 5386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:28.221573 5386 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:28.222749 5386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:28.227293 5386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1943df213b894a03b4fd5bdd22730d12: Generated new TSK 0
I20260925 00:57:28.228016 5386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:28.235116 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5299
2026-09-25T00:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:28 dist-test-slave-rpk1 krb5kdc[5398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:28 dist-test-slave-rpk1 krb5kdc[5398](info): set up 2 sockets
Sep 25 00:57:28 dist-test-slave-rpk1 krb5kdc[5398](info): commencing operation
krb5kdc: starting...
W20260925 00:57:29.189059 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:29 dist-test-slave-rpk1 krb5kdc[5398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:29.312047 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:29.425814 5411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:29.426126 5411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:29.426170 5411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:29.430364 5411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:29.430451 5411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:29.430471 5411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:29.430487 5411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:29.430501 5411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:29.436064 5411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:29.437341 5411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:29.438400 5411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:29.447520 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:29.447496 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:29.447552 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:29.447598 5411 server_base.cc:1034] running on GCE node
I20260925 00:57:29.448737 5411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:29.450011 5411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:29.451211 5411 hybrid_clock.cc:648] HybridClock initialized: now 1790297849451191 us; error 41 us; skew 500 ppm
Sep 25 00:57:29 dist-test-slave-rpk1 krb5kdc[5398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:29.454197 5411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:29.455215 5411 webserver.cc:460] Webserver started at http://127.0.0.1:35965/ using document root <none> and password file <none>
I20260925 00:57:29.455806 5411 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:29.455876 5411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:29.456089 5411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:29.458254 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "faef3fc353d84e28b7382e472a1d45a8"
format_stamp: "Formatted at 2026-09-25 00:57:29 on dist-test-slave-rpk1"
server_key: "3f11733b6a0b05c5e0972966bf9f7196"
server_key_iv: "f00828fbf9340d1ee608b6612a6c8def"
server_key_version: "encryptionkey@0"
I20260925 00:57:29.458827 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "faef3fc353d84e28b7382e472a1d45a8"
format_stamp: "Formatted at 2026-09-25 00:57:29 on dist-test-slave-rpk1"
server_key: "3f11733b6a0b05c5e0972966bf9f7196"
server_key_iv: "f00828fbf9340d1ee608b6612a6c8def"
server_key_version: "encryptionkey@0"
I20260925 00:57:29.462735 5411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:29.465309 5426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:29.466621 5411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:57:29.466740 5411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "faef3fc353d84e28b7382e472a1d45a8"
format_stamp: "Formatted at 2026-09-25 00:57:29 on dist-test-slave-rpk1"
server_key: "3f11733b6a0b05c5e0972966bf9f7196"
server_key_iv: "f00828fbf9340d1ee608b6612a6c8def"
server_key_version: "encryptionkey@0"
I20260925 00:57:29.466843 5411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:29.499418 5411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:29.502660 5411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:29.502847 5411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:29.510054 5411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39297
I20260925 00:57:29.510082 5478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39297 every 8 connection(s)
I20260925 00:57:29.511238 5411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:29.514117 5479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:29.519188 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5411
I20260925 00:57:29.519322 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:29.519639 17192 external_mini_cluster.cc:1428] Setting key 153b591140212fefcabd034c95b55bbc
I20260925 00:57:29.519917 5479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8: Bootstrap starting.
I20260925 00:57:29.522646 5479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:29.523524 5479 log.cc:824] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:29.525651 5479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8: No bootstrap required, opened a new log
I20260925 00:57:29.529899 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:29.521117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51009 (local address 127.0.0.1:39297)
0925 00:57:29.521555 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:29.521574 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:29.521617 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:57:29.522466 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:29.522490 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:29.522949 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:29.523307 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:29.523749 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:29.525090 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:29.526041 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:29.526739 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:29.526900 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:29.527695 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:29.527702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:29.527719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:29.528274 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:29.528364 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:29.528398 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:29.528504 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:29.528647 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 00:57:29.528892 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260925 00:57:29.529917 5479 raft_consensus.cc:348] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faef3fc353d84e28b7382e472a1d45a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260925 00:57:29.530246 5479 raft_consensus.cc:374] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:29.530303 5479 raft_consensus.cc:729] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faef3fc353d84e28b7382e472a1d45a8, State: Initialized, Role: FOLLOWER
I20260925 00:57:29.530716 5479 consensus_queue.cc:260] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faef3fc353d84e28b7382e472a1d45a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260925 00:57:29.530854 5479 raft_consensus.cc:388] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:29.530920 5479 raft_consensus.cc:482] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:29.531016 5479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:29.531980 5479 raft_consensus.cc:504] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faef3fc353d84e28b7382e472a1d45a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260925 00:57:29.532321 5479 leader_election.cc:302] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faef3fc353d84e28b7382e472a1d45a8; no voters:
I20260925 00:57:29.532634 5479 leader_election.cc:288] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:29.532758 5484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:29.532856 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:29.533031 5484 raft_consensus.cc:686] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [term 1 LEADER]: Becoming Leader. State: Replica: faef3fc353d84e28b7382e472a1d45a8, State: Running, Role: LEADER
I20260925 00:57:29.533401 5484 consensus_queue.cc:237] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faef3fc353d84e28b7382e472a1d45a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260925 00:57:29.533949 5479 sys_catalog.cc:564] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:29.535079 5485 sys_catalog.cc:455] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faef3fc353d84e28b7382e472a1d45a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faef3fc353d84e28b7382e472a1d45a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } } }
I20260925 00:57:29.535223 5485 sys_catalog.cc:458] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:29.535471 5485 sys_catalog.cc:455] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faef3fc353d84e28b7382e472a1d45a8. Latest consensus state: current_term: 1 leader_uuid: "faef3fc353d84e28b7382e472a1d45a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faef3fc353d84e28b7382e472a1d45a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } } }
I20260925 00:57:29.535549 5485 sys_catalog.cc:458] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:29.535921 5498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:29.539472 5498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:29.539811 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:29.535441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55939 (local address 127.0.0.1:39297)
0925 00:57:29.535556 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:29.535561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:29.535713 (+ 152us) server_negotiation.cc:407] Connection header received
0925 00:57:29.535861 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:29.535864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:29.535914 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:29.536005 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:29.536306 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:29.537088 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:29.538049 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:29.538378 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:29.538482 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:29.539335 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:29.539339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:29.539342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:29.539355 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:29.539384 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:29.539398 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:29.539476 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:29.539613 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 00:57:29.539675 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 00:57:29.545547 5498 catalog_manager.cc:1347] Generated new cluster ID: 25b205f8ec5d47c5b0049ae9e84ab588
I20260925 00:57:29.545609 5498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:29.566859 5498 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:29.567879 5498 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:29.573990 5498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faef3fc353d84e28b7382e472a1d45a8: Generated new TSK 0
I20260925 00:57:29.574748 5498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:29.580200 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5411
2026-09-25T00:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:29 dist-test-slave-rpk1 krb5kdc[5510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:29 dist-test-slave-rpk1 krb5kdc[5510](info): set up 2 sockets
Sep 25 00:57:29 dist-test-slave-rpk1 krb5kdc[5510](info): commencing operation
krb5kdc: starting...
W20260925 00:57:30.537014 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:30 dist-test-slave-rpk1 krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:30.661553 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:30.772805 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:30.773118 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:30.773162 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:30.777262 5523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:30.777336 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:30.777355 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:30.777371 5523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:30.777385 5523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:30.782904 5523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:30.784009 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:30.785120 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:30.793823 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:30.793825 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:30.793825 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:30.794203 5523 server_base.cc:1034] running on GCE node
I20260925 00:57:30.794818 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:30.795955 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:30.797113 5523 hybrid_clock.cc:648] HybridClock initialized: now 1790297850797090 us; error 35 us; skew 500 ppm
Sep 25 00:57:30 dist-test-slave-rpk1 krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:30.799815 5523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:30.800800 5523 webserver.cc:460] Webserver started at http://127.0.0.1:37373/ using document root <none> and password file <none>
I20260925 00:57:30.801349 5523 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:30.801417 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:30.801681 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:30.803933 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "9193d1bcab074c80b1d5b1b9c8c33852"
format_stamp: "Formatted at 2026-09-25 00:57:30 on dist-test-slave-rpk1"
server_key: "e91081a356f312480f551bcea7e69e99"
server_key_iv: "18ba2b5645810b0a8c502161752deeb0"
server_key_version: "encryptionkey@0"
I20260925 00:57:30.804491 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "9193d1bcab074c80b1d5b1b9c8c33852"
format_stamp: "Formatted at 2026-09-25 00:57:30 on dist-test-slave-rpk1"
server_key: "e91081a356f312480f551bcea7e69e99"
server_key_iv: "18ba2b5645810b0a8c502161752deeb0"
server_key_version: "encryptionkey@0"
I20260925 00:57:30.808523 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 00:57:30.811093 5538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:30.812469 5523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:57:30.812656 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "9193d1bcab074c80b1d5b1b9c8c33852"
format_stamp: "Formatted at 2026-09-25 00:57:30 on dist-test-slave-rpk1"
server_key: "e91081a356f312480f551bcea7e69e99"
server_key_iv: "18ba2b5645810b0a8c502161752deeb0"
server_key_version: "encryptionkey@0"
I20260925 00:57:30.812858 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:30.817629 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:30.821030 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:30.821218 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:30.828156 5523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42921
I20260925 00:57:30.828187 5590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42921 every 8 connection(s)
I20260925 00:57:30.829229 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:30.832175 5591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:30.837767 5591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852: Bootstrap starting.
I20260925 00:57:30.838359 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5523
I20260925 00:57:30.838495 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:30.838779 17192 external_mini_cluster.cc:1428] Setting key c33aab897cd93862257f31e48dccb4b3
I20260925 00:57:30.840701 5591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:30.841454 5591 log.cc:824] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:30.843502 5591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852: No bootstrap required, opened a new log
I20260925 00:57:30.844059 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:30.840162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46809 (local address 127.0.0.1:42921)
0925 00:57:30.840604 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:30.840618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:30.840654 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:57:30.841372 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:30.841390 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:30.841741 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:30.842066 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:30.842805 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:30.842810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:30.842822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:30.842832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:30.842882 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:30.842900 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:30.842995 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:30.843118 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 00:57:30.843276 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260925 00:57:30.846906 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:30.847221 5591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42921 } }
I20260925 00:57:30.847464 5591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:30.847532 5591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9193d1bcab074c80b1d5b1b9c8c33852, State: Initialized, Role: FOLLOWER
I20260925 00:57:30.847957 5591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42921 } }
I20260925 00:57:30.848089 5591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:30.848191 5591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:30.848299 5591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:30.849632 5591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42921 } }
I20260925 00:57:30.850015 5591 leader_election.cc:302] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9193d1bcab074c80b1d5b1b9c8c33852; no voters:
I20260925 00:57:30.850404 5591 leader_election.cc:288] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:30.850494 5601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:30.850626 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:30.849004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52869 (local address 127.0.0.1:42921)
0925 00:57:30.849115 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:30.849118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:30.849173 (+ 55us) server_negotiation.cc:407] Connection header received
0925 00:57:30.849313 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:30.849315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:30.849345 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:30.849410 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:30.850186 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:30.850190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:30.850193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:30.850205 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:30.850235 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:30.850248 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:30.850321 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:30.850437 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 00:57:30.850476 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:57:30.850765 5601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [term 1 LEADER]: Becoming Leader. State: Replica: 9193d1bcab074c80b1d5b1b9c8c33852, State: Running, Role: LEADER
I20260925 00:57:30.851150 5601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42921 } }
I20260925 00:57:30.851774 5591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:30.852340 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9193d1bcab074c80b1d5b1b9c8c33852. Latest consensus state: current_term: 1 leader_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42921 } } }
I20260925 00:57:30.852460 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:30.852928 5608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:30.853154 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193d1bcab074c80b1d5b1b9c8c33852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42921 } } }
I20260925 00:57:30.853318 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:30.856310 5608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:30.862568 5608 catalog_manager.cc:1347] Generated new cluster ID: 7a4f92a48d41481591a5979ab93791ef
I20260925 00:57:30.862660 5608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:30.887049 5608 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:30.887431 5608 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:30.894069 5608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9193d1bcab074c80b1d5b1b9c8c33852: Generated new TSK 0
I20260925 00:57:30.894917 5608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:30.905930 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5523
2026-09-25T00:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:30 dist-test-slave-rpk1 krb5kdc[5622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:30 dist-test-slave-rpk1 krb5kdc[5622](info): set up 2 sockets
Sep 25 00:57:30 dist-test-slave-rpk1 krb5kdc[5622](info): commencing operation
krb5kdc: starting...
W20260925 00:57:31.876792 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:31 dist-test-slave-rpk1 krb5kdc[5622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:32.000435 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:32.117820 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:32.118145 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:32.118191 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:32.122463 5635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:32.122545 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:32.122563 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:32.122578 5635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:32.122593 5635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:32.128204 5635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:32.129369 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:32.130421 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:57:32.140162 5635 server_base.cc:1034] running on GCE node
W20260925 00:57:32.140118 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:32.140133 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:32.140118 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:32.141108 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:32.142306 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:32.143545 5635 hybrid_clock.cc:648] HybridClock initialized: now 1790297852143521 us; error 43 us; skew 500 ppm
Sep 25 00:57:32 dist-test-slave-rpk1 krb5kdc[5622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:32.146541 5635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:32.147454 5635 webserver.cc:460] Webserver started at http://127.0.0.1:38853/ using document root <none> and password file <none>
I20260925 00:57:32.148005 5635 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:32.148074 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:32.148276 5635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:32.150426 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "a676729fe2434e4b8755124402f4ce7b"
format_stamp: "Formatted at 2026-09-25 00:57:32 on dist-test-slave-rpk1"
server_key: "57d29c720adbaa3837487a8c2706441e"
server_key_iv: "23f061db160a7d5f5c370dd1990bcded"
server_key_version: "encryptionkey@0"
I20260925 00:57:32.150975 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "a676729fe2434e4b8755124402f4ce7b"
format_stamp: "Formatted at 2026-09-25 00:57:32 on dist-test-slave-rpk1"
server_key: "57d29c720adbaa3837487a8c2706441e"
server_key_iv: "23f061db160a7d5f5c370dd1990bcded"
server_key_version: "encryptionkey@0"
I20260925 00:57:32.154873 5635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 00:57:32.157418 5650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:32.158790 5635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 00:57:32.158932 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "a676729fe2434e4b8755124402f4ce7b"
format_stamp: "Formatted at 2026-09-25 00:57:32 on dist-test-slave-rpk1"
server_key: "57d29c720adbaa3837487a8c2706441e"
server_key_iv: "23f061db160a7d5f5c370dd1990bcded"
server_key_version: "encryptionkey@0"
I20260925 00:57:32.159047 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:32.186892 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:32.190390 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:32.190588 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:32.197784 5635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40695
I20260925 00:57:32.197804 5702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40695 every 8 connection(s)
I20260925 00:57:32.198796 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:32.201610 5703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:32.207269 5703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b: Bootstrap starting.
I20260925 00:57:32.207525 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5635
I20260925 00:57:32.207664 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:32.207930 17192 external_mini_cluster.cc:1428] Setting key 7df8b65820f180121d6250a60d2c6e34
I20260925 00:57:32.209930 5703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:32.210687 5703 log.cc:824] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:32.213053 5703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b: No bootstrap required, opened a new log
I20260925 00:57:32.217425 5703 raft_consensus.cc:348] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a676729fe2434e4b8755124402f4ce7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260925 00:57:32.217675 5703 raft_consensus.cc:374] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:32.217718 5703 raft_consensus.cc:729] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a676729fe2434e4b8755124402f4ce7b, State: Initialized, Role: FOLLOWER
Sep 25 00:57:32 dist-test-slave-rpk1 krb5kdc[5622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:57:32.218200 5703 consensus_queue.cc:260] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a676729fe2434e4b8755124402f4ce7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260925 00:57:32.218364 5703 raft_consensus.cc:388] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:32.218433 5703 raft_consensus.cc:482] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:32.218549 5703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:32.219477 5703 raft_consensus.cc:504] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a676729fe2434e4b8755124402f4ce7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260925 00:57:32.219818 5703 leader_election.cc:302] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a676729fe2434e4b8755124402f4ce7b; no voters:
I20260925 00:57:32.220144 5703 leader_election.cc:288] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:32.220237 5708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:32.220397 5708 raft_consensus.cc:686] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [term 1 LEADER]: Becoming Leader. State: Replica: a676729fe2434e4b8755124402f4ce7b, State: Running, Role: LEADER
I20260925 00:57:32.220809 5708 consensus_queue.cc:237] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a676729fe2434e4b8755124402f4ce7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260925 00:57:32.221534 5703 sys_catalog.cc:564] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:32.222565 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a676729fe2434e4b8755124402f4ce7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a676729fe2434e4b8755124402f4ce7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260925 00:57:32.222728 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:32.223073 5710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a676729fe2434e4b8755124402f4ce7b. Latest consensus state: current_term: 1 leader_uuid: "a676729fe2434e4b8755124402f4ce7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a676729fe2434e4b8755124402f4ce7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260925 00:57:32.223174 5710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:32.224436 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:32.209455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:56229 (local address 127.0.0.1:40695)
0925 00:57:32.209908 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:32.209923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:32.209962 (+ 39us) server_negotiation.cc:407] Connection header received
0925 00:57:32.210709 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:32.210730 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:32.211168 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:32.211566 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:32.212527 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:32.213616 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:32.214802 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:32.215289 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:32.215411 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:32.218426 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:32.218433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:32.218453 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:32.218491 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:32.220345 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:32.221047 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:32.221052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:32.221057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:32.221167 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:32.221496 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:32.221500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:32.221502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:32.221985 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:32.222889 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:32.223061 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 00:57:32.223290 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":134,"threads_started":1}
I20260925 00:57:32.224467 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:32.227702 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:32.228034 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:32.234187 5717 catalog_manager.cc:1347] Generated new cluster ID: eb5b8f5ec71645d6ad864698e8b1df67
I20260925 00:57:32.234277 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:32.242383 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:32.230497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:42301 (local address 127.0.0.1:40695)
0925 00:57:32.230756 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:32.230761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:32.230791 (+ 30us) server_negotiation.cc:407] Connection header received
0925 00:57:32.230938 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:32.230942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:32.230994 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:32.231068 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:32.231796 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:32.232928 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:32.236533 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:32.236972 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:32.237088 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:32.238851 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:32.238855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:32.238858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:32.238886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:32.240539 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:32.241109 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:32.241113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:32.241115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:32.241198 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:32.241513 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:32.241516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:32.241518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:32.241720 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:32.241867 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:32.242114 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 00:57:32.242215 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260925 00:57:32.267529 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:32.268913 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:32.279843 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a676729fe2434e4b8755124402f4ce7b: Generated new TSK 0
I20260925 00:57:32.280874 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:32.286756 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5635
W20260925 00:57:32.291677 5620 connection.cc:535] client connection to 127.0.0.1:40695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40695): Connection reset by peer (error 104)
2026-09-25T00:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:32 dist-test-slave-rpk1 krb5kdc[5734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:32 dist-test-slave-rpk1 krb5kdc[5734](info): set up 2 sockets
Sep 25 00:57:32 dist-test-slave-rpk1 krb5kdc[5734](info): commencing operation
krb5kdc: starting...
W20260925 00:57:33.257027 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:33 dist-test-slave-rpk1 krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:33.379359 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:33.489598 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:33.489892 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:33.489934 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:33.494029 5747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:33.494096 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:33.494115 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:33.494131 5747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:33.494151 5747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:33.499454 5747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:33.500468 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:33.501487 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:33.510010 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:33.510021 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:33.510088 5747 server_base.cc:1034] running on GCE node
W20260925 00:57:33.510010 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:33.510839 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:33.511947 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:33.513118 5747 hybrid_clock.cc:648] HybridClock initialized: now 1790297853513098 us; error 30 us; skew 500 ppm
Sep 25 00:57:33 dist-test-slave-rpk1 krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:33.521091 5747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:33.522032 5747 webserver.cc:460] Webserver started at http://127.0.0.1:46191/ using document root <none> and password file <none>
I20260925 00:57:33.522758 5747 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:33.522850 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:33.523118 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:33.525686 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "14f4cc5e243e4b61b0574993dbd20d2e"
format_stamp: "Formatted at 2026-09-25 00:57:33 on dist-test-slave-rpk1"
server_key: "09685f7230853df9353bbb64125a3918"
server_key_iv: "4d45ed5f2f3de5325edc1d4f4740af7b"
server_key_version: "encryptionkey@0"
I20260925 00:57:33.526258 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "14f4cc5e243e4b61b0574993dbd20d2e"
format_stamp: "Formatted at 2026-09-25 00:57:33 on dist-test-slave-rpk1"
server_key: "09685f7230853df9353bbb64125a3918"
server_key_iv: "4d45ed5f2f3de5325edc1d4f4740af7b"
server_key_version: "encryptionkey@0"
I20260925 00:57:33.529990 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:57:33.532562 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:33.533944 5747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:33.534060 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "14f4cc5e243e4b61b0574993dbd20d2e"
format_stamp: "Formatted at 2026-09-25 00:57:33 on dist-test-slave-rpk1"
server_key: "09685f7230853df9353bbb64125a3918"
server_key_iv: "4d45ed5f2f3de5325edc1d4f4740af7b"
server_key_version: "encryptionkey@0"
I20260925 00:57:33.534173 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:33.555073 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:33.558228 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:33.558424 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:33.565351 5747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45815
I20260925 00:57:33.565358 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45815 every 8 connection(s)
I20260925 00:57:33.566313 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:33.569088 5815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:33.574661 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e: Bootstrap starting.
I20260925 00:57:33.576156 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5747
I20260925 00:57:33.576304 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:33.576552 17192 external_mini_cluster.cc:1428] Setting key 234275581aaf17d31f11914e38701332
I20260925 00:57:33.577381 5815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:33.578225 5815 log.cc:824] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:33.580524 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e: No bootstrap required, opened a new log
I20260925 00:57:33.584813 5815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260925 00:57:33.585145 5815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:33.585209 5815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f4cc5e243e4b61b0574993dbd20d2e, State: Initialized, Role: FOLLOWER
I20260925 00:57:33.585712 5815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260925 00:57:33.585927 5815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:33.585999 5815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:33.586091 5815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:33.586879 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:33.577846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:48179 (local address 127.0.0.1:45815)
0925 00:57:33.578308 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:33.578329 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:33.578372 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:57:33.579128 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:33.579153 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:33.579600 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:33.579982 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:33.580409 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:33.581867 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:33.582865 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:33.583637 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:33.583797 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:33.584747 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:33.584753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:33.584769 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:33.585303 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:33.585378 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:33.585409 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:33.585543 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:33.585750 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 00:57:33.585897 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260925 00:57:33.587256 5815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260925 00:57:33.587591 5815 leader_election.cc:302] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f4cc5e243e4b61b0574993dbd20d2e; no voters:
I20260925 00:57:33.587888 5815 leader_election.cc:288] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:33.587978 5820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:33.588169 5820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [term 1 LEADER]: Becoming Leader. State: Replica: 14f4cc5e243e4b61b0574993dbd20d2e, State: Running, Role: LEADER
I20260925 00:57:33.588538 5820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } }
I20260925 00:57:33.589253 5815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:33.589745 5822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } } }
I20260925 00:57:33.589871 5822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:33.590128 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:33.590518 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14f4cc5e243e4b61b0574993dbd20d2e. Latest consensus state: current_term: 1 leader_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4cc5e243e4b61b0574993dbd20d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45815 } } }
I20260925 00:57:33.590607 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:33.597746 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:33.592247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:42717 (local address 127.0.0.1:45815)
0925 00:57:33.592363 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:33.592367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:33.592571 (+ 204us) server_negotiation.cc:407] Connection header received
0925 00:57:33.592660 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:33.592664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:33.592746 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:33.592825 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:33.593064 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:33.593888 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:33.595080 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:33.596197 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:33.596299 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:33.597280 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:33.597284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:33.597287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:33.597300 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:33.597328 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:33.597341 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:33.597440 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:33.597579 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 00:57:33.597612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 00:57:33.598371 5841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:33.598448 5841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:33.598505 5825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:33.599247 5825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:33.604972 5825 catalog_manager.cc:1347] Generated new cluster ID: 543f68b4ba2a41ca910ef08495b33a5f
I20260925 00:57:33.605072 5825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:33.614665 5825 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:33.615875 5825 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:33.625682 5825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14f4cc5e243e4b61b0574993dbd20d2e: Generated new TSK 0
I20260925 00:57:33.626405 5825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:33.639123 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5747
W20260925 00:57:33.643774 5732 connection.cc:535] client connection to 127.0.0.1:45815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45815): Connection reset by peer (error 104)
2026-09-25T00:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:33 dist-test-slave-rpk1 krb5kdc[5846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:33 dist-test-slave-rpk1 krb5kdc[5846](info): set up 2 sockets
Sep 25 00:57:33 dist-test-slave-rpk1 krb5kdc[5846](info): commencing operation
krb5kdc: starting...
W20260925 00:57:34.596926 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:34 dist-test-slave-rpk1 krb5kdc[5846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:34.725474 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:34.837720 5859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:34.838038 5859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:34.838083 5859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:34.842265 5859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:34.842341 5859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:34.842360 5859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:34.842376 5859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:34.842391 5859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:34.848045 5859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:34.849543 5859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:34.850698 5859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:34.860042 5864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:34.860042 5865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:34.860036 5867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:34.860252 5859 server_base.cc:1034] running on GCE node
I20260925 00:57:34.861088 5859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:34.862277 5859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:34.863485 5859 hybrid_clock.cc:648] HybridClock initialized: now 1790297854863460 us; error 41 us; skew 500 ppm
Sep 25 00:57:34 dist-test-slave-rpk1 krb5kdc[5846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:34.866365 5859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:34.867324 5859 webserver.cc:460] Webserver started at http://127.0.0.1:44875/ using document root <none> and password file <none>
I20260925 00:57:34.867865 5859 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:34.867926 5859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:34.868095 5859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:34.870258 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "93ae5f0dded9493783e6784a54b5f7a4"
format_stamp: "Formatted at 2026-09-25 00:57:34 on dist-test-slave-rpk1"
server_key: "dca39b5b1ef67717489977d37b792e9b"
server_key_iv: "e80deb2a3cf9151089d728dc6d9ae905"
server_key_version: "encryptionkey@0"
I20260925 00:57:34.870800 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "93ae5f0dded9493783e6784a54b5f7a4"
format_stamp: "Formatted at 2026-09-25 00:57:34 on dist-test-slave-rpk1"
server_key: "dca39b5b1ef67717489977d37b792e9b"
server_key_iv: "e80deb2a3cf9151089d728dc6d9ae905"
server_key_version: "encryptionkey@0"
I20260925 00:57:34.874818 5859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:34.877481 5874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:34.878916 5859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:34.879060 5859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "93ae5f0dded9493783e6784a54b5f7a4"
format_stamp: "Formatted at 2026-09-25 00:57:34 on dist-test-slave-rpk1"
server_key: "dca39b5b1ef67717489977d37b792e9b"
server_key_iv: "e80deb2a3cf9151089d728dc6d9ae905"
server_key_version: "encryptionkey@0"
I20260925 00:57:34.879172 5859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:34.883931 5859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:34.887115 5859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:34.887288 5859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:34.894215 5859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45803
I20260925 00:57:34.894239 5926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45803 every 8 connection(s)
I20260925 00:57:34.895176 5859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:34.898155 5927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:34.902244 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5859
I20260925 00:57:34.902423 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:34.902686 17192 external_mini_cluster.cc:1428] Setting key f689b17134dc5d3d62b35df9515304b1
I20260925 00:57:34.904024 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4: Bootstrap starting.
I20260925 00:57:34.906733 5927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:34.907606 5927 log.cc:824] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:34.908535 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:34.904165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:36379 (local address 127.0.0.1:45803)
0925 00:57:34.904580 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:34.904599 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:34.904640 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:57:34.905497 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:34.905521 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:34.905888 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:34.906253 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:34.907099 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:34.907105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:34.907121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:34.907133 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:34.907194 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:34.907220 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:34.907326 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:34.907505 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 00:57:34.907670 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
I20260925 00:57:34.909783 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4: No bootstrap required, opened a new log
I20260925 00:57:34.911497 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:34.914016 5927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ae5f0dded9493783e6784a54b5f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260925 00:57:34.914320 5927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:34.914397 5927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ae5f0dded9493783e6784a54b5f7a4, State: Initialized, Role: FOLLOWER
I20260925 00:57:34.914889 5927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ae5f0dded9493783e6784a54b5f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260925 00:57:34.915066 5927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:34.915143 5927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:34.915244 5927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:34.915673 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:34.913886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:49149 (local address 127.0.0.1:45803)
0925 00:57:34.914019 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:34.914024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:34.914186 (+ 162us) server_negotiation.cc:407] Connection header received
0925 00:57:34.914317 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:34.914320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:34.914372 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:34.914465 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:34.915252 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:34.915255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:34.915258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:34.915269 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:34.915346 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:34.915362 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:34.915443 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:34.915514 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 00:57:34.915555 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:34.916265 5927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ae5f0dded9493783e6784a54b5f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260925 00:57:34.916600 5927 leader_election.cc:302] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ae5f0dded9493783e6784a54b5f7a4; no voters:
I20260925 00:57:34.916908 5927 leader_election.cc:288] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:34.917093 5937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:34.917294 5937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [term 1 LEADER]: Becoming Leader. State: Replica: 93ae5f0dded9493783e6784a54b5f7a4, State: Running, Role: LEADER
I20260925 00:57:34.917673 5937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ae5f0dded9493783e6784a54b5f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260925 00:57:34.918341 5927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:34.919265 5938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93ae5f0dded9493783e6784a54b5f7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ae5f0dded9493783e6784a54b5f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } } }
I20260925 00:57:34.919385 5938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:34.919679 5939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ae5f0dded9493783e6784a54b5f7a4. Latest consensus state: current_term: 1 leader_uuid: "93ae5f0dded9493783e6784a54b5f7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ae5f0dded9493783e6784a54b5f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } } }
I20260925 00:57:34.919770 5939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:34.919788 5946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:34.923187 5946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:34.929316 5946 catalog_manager.cc:1347] Generated new cluster ID: e1d9e501e66b4f17a08fa9c8fe4e5fcb
I20260925 00:57:34.929482 5946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:34.948585 5946 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:34.949024 5946 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:34.960266 5946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93ae5f0dded9493783e6784a54b5f7a4: Generated new TSK 0
I20260925 00:57:34.961210 5946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:34.965598 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5859
2026-09-25T00:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:35 dist-test-slave-rpk1 krb5kdc[5958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:35 dist-test-slave-rpk1 krb5kdc[5958](info): set up 2 sockets
Sep 25 00:57:35 dist-test-slave-rpk1 krb5kdc[5958](info): commencing operation
krb5kdc: starting...
W20260925 00:57:35.932924 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:36 dist-test-slave-rpk1 krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:36.060782 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:36.172712 5971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:36.173023 5971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:36.173084 5971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:36.177244 5971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:36.177317 5971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:36.177345 5971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:36.177366 5971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:36.177392 5971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:36.182900 5971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44301
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:36.184185 5971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:36.185261 5971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:36.193889 5976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:36.193889 5977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:36.193894 5979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:36.194228 5971 server_base.cc:1034] running on GCE node
I20260925 00:57:36.194819 5971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:36.196182 5971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:36.197371 5971 hybrid_clock.cc:648] HybridClock initialized: now 1790297856197328 us; error 50 us; skew 500 ppm
Sep 25 00:57:36 dist-test-slave-rpk1 krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:36.200363 5971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:36.201529 5971 webserver.cc:460] Webserver started at http://127.0.0.1:39051/ using document root <none> and password file <none>
I20260925 00:57:36.202119 5971 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:36.202191 5971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:36.202414 5971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:36.204753 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "81d7ce52b16f414984e2583e50cd2fd9"
format_stamp: "Formatted at 2026-09-25 00:57:36 on dist-test-slave-rpk1"
server_key: "4805ff266c855ebf9e01e60ffaa8422a"
server_key_iv: "a0e5542cb847f325d731ac3acc30bf0e"
server_key_version: "encryptionkey@0"
I20260925 00:57:36.205356 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "81d7ce52b16f414984e2583e50cd2fd9"
format_stamp: "Formatted at 2026-09-25 00:57:36 on dist-test-slave-rpk1"
server_key: "4805ff266c855ebf9e01e60ffaa8422a"
server_key_iv: "a0e5542cb847f325d731ac3acc30bf0e"
server_key_version: "encryptionkey@0"
I20260925 00:57:36.210090 5971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 00:57:36.213017 5986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:36.214430 5971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:57:36.214584 5971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "81d7ce52b16f414984e2583e50cd2fd9"
format_stamp: "Formatted at 2026-09-25 00:57:36 on dist-test-slave-rpk1"
server_key: "4805ff266c855ebf9e01e60ffaa8422a"
server_key_iv: "a0e5542cb847f325d731ac3acc30bf0e"
server_key_version: "encryptionkey@0"
I20260925 00:57:36.214715 5971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:36.242983 5971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:36.246430 5971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:36.246656 5971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:36.255247 5971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44301
I20260925 00:57:36.255283 6038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44301 every 8 connection(s)
I20260925 00:57:36.256628 5971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:36.257758 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 5971
I20260925 00:57:36.257889 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:36.258181 17192 external_mini_cluster.cc:1428] Setting key 622fd50c46af7495b42bcc25d0826800
I20260925 00:57:36.260020 6039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:36.267832 6039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9: Bootstrap starting.
Sep 25 00:57:36 dist-test-slave-rpk1 krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:57:36.270862 6039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:36.271838 6039 log.cc:824] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:36.274327 6039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9: No bootstrap required, opened a new log
I20260925 00:57:36.276844 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:36.259902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:37907 (local address 127.0.0.1:44301)
0925 00:57:36.260488 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:36.260508 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:36.260551 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:57:36.261494 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:36.261520 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:36.262037 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:36.262456 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:36.263402 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:36.265074 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:36.266302 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:36.267068 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:36.267273 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:36.270350 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:36.270359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:36.270380 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:36.270420 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:36.272648 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:36.273305 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:36.273312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:36.273319 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:36.273480 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:36.273902 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:36.273906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:36.273908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:36.274553 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:36.274774 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:36.275487 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 00:57:36.275740 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":130,"threads_started":1}
I20260925 00:57:36.278844 6039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d7ce52b16f414984e2583e50cd2fd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44301 } }
I20260925 00:57:36.279162 6039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:36.279256 6039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d7ce52b16f414984e2583e50cd2fd9, State: Initialized, Role: FOLLOWER
I20260925 00:57:36.279737 6039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d7ce52b16f414984e2583e50cd2fd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44301 } }
I20260925 00:57:36.279928 6039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:36.280016 6039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:36.280150 6039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:36.280833 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:36.281459 6039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d7ce52b16f414984e2583e50cd2fd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44301 } }
I20260925 00:57:36.281852 6039 leader_election.cc:302] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d7ce52b16f414984e2583e50cd2fd9; no voters:
I20260925 00:57:36.282197 6039 leader_election.cc:288] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:36.282331 6044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:36.282549 6044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [term 1 LEADER]: Becoming Leader. State: Replica: 81d7ce52b16f414984e2583e50cd2fd9, State: Running, Role: LEADER
I20260925 00:57:36.282888 6044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d7ce52b16f414984e2583e50cd2fd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44301 } }
I20260925 00:57:36.283588 6039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:36.284265 6044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d7ce52b16f414984e2583e50cd2fd9. Latest consensus state: current_term: 1 leader_uuid: "81d7ce52b16f414984e2583e50cd2fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d7ce52b16f414984e2583e50cd2fd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44301 } } }
I20260925 00:57:36.284416 6044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:36.284797 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d7ce52b16f414984e2583e50cd2fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d7ce52b16f414984e2583e50cd2fd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44301 } } }
I20260925 00:57:36.284966 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9 [sys.catalog]: This master's current role is: LEADER
W20260925 00:57:36.289860 6065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:36.290026 6065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:36.290170 6053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:36.291153 6053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:36.294667 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:36.283701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:34595 (local address 127.0.0.1:44301)
0925 00:57:36.283814 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:36.283818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:36.283835 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:57:36.283873 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:36.283876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:36.283919 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:36.283986 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:36.286533 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:36.287303 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:36.288244 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:36.288590 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:36.288707 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:36.290690 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:36.290695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:36.290698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:36.290725 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:36.292382 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:36.293047 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:36.293051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:36.293053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:36.293143 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:36.293582 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:36.293585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:36.293587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:36.293816 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:36.293948 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:36.294379 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 00:57:36.294486 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:57:36.298688 6053 catalog_manager.cc:1347] Generated new cluster ID: 033da80551fa44caa43b26edbd4b15b1
I20260925 00:57:36.298787 6053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:36.326858 6053 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:36.328116 6053 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:36.334487 6053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d7ce52b16f414984e2583e50cd2fd9: Generated new TSK 0
I20260925 00:57:36.335183 6053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:36.342253 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 5971
W20260925 00:57:36.347002 5956 connection.cc:535] client connection to 127.0.0.1:44301 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44301): Connection reset by peer (error 104)
2026-09-25T00:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:36 dist-test-slave-rpk1 krb5kdc[6070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:36 dist-test-slave-rpk1 krb5kdc[6070](info): set up 2 sockets
Sep 25 00:57:36 dist-test-slave-rpk1 krb5kdc[6070](info): commencing operation
krb5kdc: starting...
W20260925 00:57:37.305150 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:37 dist-test-slave-rpk1 krb5kdc[6070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:37.432284 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:37.546275 6083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:37.546588 6083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:37.546651 6083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:37.550891 6083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:37.550973 6083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:37.551002 6083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:37.551026 6083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:37.551056 6083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:37.557392 6083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:37.558581 6083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:37.559645 6083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:57:37.568495 6083 server_base.cc:1034] running on GCE node
W20260925 00:57:37.568542 6089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:37.568554 6088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:37.568784 6091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:37.569355 6083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:37.570564 6083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:37.571760 6083 hybrid_clock.cc:648] HybridClock initialized: now 1790297857571732 us; error 39 us; skew 500 ppm
Sep 25 00:57:37 dist-test-slave-rpk1 krb5kdc[6070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:37.574863 6083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:37.575850 6083 webserver.cc:460] Webserver started at http://127.0.0.1:36815/ using document root <none> and password file <none>
I20260925 00:57:37.576393 6083 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:37.576463 6083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:37.576642 6083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:37.578912 6083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "f62cd1df86f24c9aa1ffdaa6604565f7"
format_stamp: "Formatted at 2026-09-25 00:57:37 on dist-test-slave-rpk1"
server_key: "68faa4298e832ce4c0924b1a4d399c7a"
server_key_iv: "4e0520826c4302d03230cdf059008f8b"
server_key_version: "encryptionkey@0"
I20260925 00:57:37.579497 6083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "f62cd1df86f24c9aa1ffdaa6604565f7"
format_stamp: "Formatted at 2026-09-25 00:57:37 on dist-test-slave-rpk1"
server_key: "68faa4298e832ce4c0924b1a4d399c7a"
server_key_iv: "4e0520826c4302d03230cdf059008f8b"
server_key_version: "encryptionkey@0"
I20260925 00:57:37.583796 6083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:37.586786 6098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:37.588167 6083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:57:37.588303 6083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "f62cd1df86f24c9aa1ffdaa6604565f7"
format_stamp: "Formatted at 2026-09-25 00:57:37 on dist-test-slave-rpk1"
server_key: "68faa4298e832ce4c0924b1a4d399c7a"
server_key_iv: "4e0520826c4302d03230cdf059008f8b"
server_key_version: "encryptionkey@0"
I20260925 00:57:37.588457 6083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:37.601998 6083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:37.605367 6083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:37.605563 6083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:37.612248 6083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40973
I20260925 00:57:37.612288 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40973 every 8 connection(s)
I20260925 00:57:37.613242 6083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:37.616037 6151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:37.619074 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6083
I20260925 00:57:37.619187 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:37.619460 17192 external_mini_cluster.cc:1428] Setting key 42d08e03a4a906ceeab861306713b650
I20260925 00:57:37.622043 6151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7: Bootstrap starting.
I20260925 00:57:37.624306 6151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:37.625090 6151 log.cc:824] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:37.627018 6151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7: No bootstrap required, opened a new log
I20260925 00:57:37.628541 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:37.620861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:44959 (local address 127.0.0.1:40973)
0925 00:57:37.621353 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:37.621370 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:37.621408 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:57:37.622089 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:37.622110 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:37.622504 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:37.622849 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:37.623261 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:37.624287 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:37.625250 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:37.625881 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:37.626028 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:37.626889 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:37.626894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:37.626908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:37.627319 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:37.627376 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:37.627400 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:37.627512 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:37.627688 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 00:57:37.627834 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":149,"threads_started":1}
I20260925 00:57:37.630833 6151 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 00:57:37.631131 6151 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:37.631212 6151 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62cd1df86f24c9aa1ffdaa6604565f7, State: Initialized, Role: FOLLOWER
I20260925 00:57:37.631697 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:37.631669 6151 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 00:57:37.631803 6151 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:37.631867 6151 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:37.631974 6151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:37.632947 6151 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 00:57:37.633287 6151 leader_election.cc:302] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62cd1df86f24c9aa1ffdaa6604565f7; no voters:
I20260925 00:57:37.633579 6151 leader_election.cc:288] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:37.633684 6160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:37.634069 6160 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [term 1 LEADER]: Becoming Leader. State: Replica: f62cd1df86f24c9aa1ffdaa6604565f7, State: Running, Role: LEADER
I20260925 00:57:37.634447 6160 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 00:57:37.635087 6151 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:37.635661 6160 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62cd1df86f24c9aa1ffdaa6604565f7. Latest consensus state: current_term: 1 leader_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260925 00:57:37.636368 6160 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:37.636260 6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cd1df86f24c9aa1ffdaa6604565f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260925 00:57:37.636693 6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:37.638624 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:37.633709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:39421 (local address 127.0.0.1:40973)
0925 00:57:37.633808 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:37.633812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:37.634080 (+ 268us) server_negotiation.cc:407] Connection header received
0925 00:57:37.634224 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:37.634228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:37.634278 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:37.634325 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:37.634754 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:37.635530 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:37.636703 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:37.637007 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:37.637081 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:37.638190 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:37.638193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:37.638195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:37.638207 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:37.638231 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:37.638242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:37.638312 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:37.638493 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 00:57:37.638525 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 00:57:37.640467 6176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:37.640554 6176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:37.640698 6170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:37.641597 6170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:37.647707 6170 catalog_manager.cc:1347] Generated new cluster ID: d189f277e2ff47f59eca3cf72f841932
I20260925 00:57:37.647804 6170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:37.663036 6170 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:37.664352 6170 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:37.672443 6170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f62cd1df86f24c9aa1ffdaa6604565f7: Generated new TSK 0
I20260925 00:57:37.673559 6170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:37.688997 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6083
W20260925 00:57:37.693766 6068 connection.cc:535] client connection to 127.0.0.1:40973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40973): Connection reset by peer (error 104)
2026-09-25T00:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:37 dist-test-slave-rpk1 krb5kdc[6182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:37 dist-test-slave-rpk1 krb5kdc[6182](info): set up 2 sockets
Sep 25 00:57:37 dist-test-slave-rpk1 krb5kdc[6182](info): commencing operation
krb5kdc: starting...
W20260925 00:57:38.644974 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:38 dist-test-slave-rpk1 krb5kdc[6182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:57:38.766311 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:38.876830 6195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:38.877127 6195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:38.877170 6195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:38.881353 6195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:38.881438 6195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:38.881457 6195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:38.881472 6195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:38.881486 6195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:38.887095 6195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:38.888234 6195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:38.889312 6195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:38.898375 6200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:38.898429 6195 server_base.cc:1034] running on GCE node
W20260925 00:57:38.898384 6201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:38.898391 6203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:38.899237 6195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:38.900418 6195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:38.901598 6195 hybrid_clock.cc:648] HybridClock initialized: now 1790297858901565 us; error 48 us; skew 500 ppm
Sep 25 00:57:38 dist-test-slave-rpk1 krb5kdc[6182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:38.910506 6195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:57:38.911625 6195 webserver.cc:460] Webserver started at http://127.0.0.1:42757/ using document root <none> and password file <none>
I20260925 00:57:38.912247 6195 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:38.912326 6195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:38.912551 6195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:38.914836 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "e3398b7c4d154be09432d95ec45e1029"
format_stamp: "Formatted at 2026-09-25 00:57:38 on dist-test-slave-rpk1"
server_key: "b35b11258494cc107edbdd7a1335b3b8"
server_key_iv: "6cfb255aaf452a572349eb4df6b1553b"
server_key_version: "encryptionkey@0"
I20260925 00:57:38.915467 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "e3398b7c4d154be09432d95ec45e1029"
format_stamp: "Formatted at 2026-09-25 00:57:38 on dist-test-slave-rpk1"
server_key: "b35b11258494cc107edbdd7a1335b3b8"
server_key_iv: "6cfb255aaf452a572349eb4df6b1553b"
server_key_version: "encryptionkey@0"
I20260925 00:57:38.919592 6195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:38.922219 6210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:38.923539 6195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:57:38.923674 6195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "e3398b7c4d154be09432d95ec45e1029"
format_stamp: "Formatted at 2026-09-25 00:57:38 on dist-test-slave-rpk1"
server_key: "b35b11258494cc107edbdd7a1335b3b8"
server_key_iv: "6cfb255aaf452a572349eb4df6b1553b"
server_key_version: "encryptionkey@0"
I20260925 00:57:38.923780 6195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:38.928367 6195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:38.931509 6195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:38.931691 6195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:38.938930 6195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39029
I20260925 00:57:38.939008 6262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39029 every 8 connection(s)
I20260925 00:57:38.939963 6195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:38.942994 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6195
I20260925 00:57:38.943025 6263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:38.943133 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:38.943434 17192 external_mini_cluster.cc:1428] Setting key 99713b0faebee63a54f1f750391f9992
I20260925 00:57:38.949198 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:38.944997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:42321 (local address 127.0.0.1:39029)
0925 00:57:38.945517 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:38.945534 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:38.945573 (+ 39us) server_negotiation.cc:407] Connection header received
0925 00:57:38.946476 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:38.946497 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:38.946809 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:38.947141 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:38.947940 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:38.947945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:38.947958 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:38.947969 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:38.948025 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:38.948045 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:38.948171 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:38.948243 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 00:57:38.948411 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":140,"threads_started":1}
I20260925 00:57:38.950335 6263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029: Bootstrap starting.
I20260925 00:57:38.952246 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:38.952816 6263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:38.953552 6263 log.cc:824] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:38.955770 6263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029: No bootstrap required, opened a new log
I20260925 00:57:38.956130 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:38.954393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:47083 (local address 127.0.0.1:39029)
0925 00:57:38.954490 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:38.954494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:38.954715 (+ 221us) server_negotiation.cc:407] Connection header received
0925 00:57:38.954857 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:38.954860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:38.954893 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:38.954953 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:38.955757 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:38.955761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:38.955764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:38.955777 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:38.955803 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:38.955814 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:38.955887 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:38.955973 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 00:57:38.956014 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 00:57:38.959524 6263 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3398b7c4d154be09432d95ec45e1029" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260925 00:57:38.959766 6263 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:38.959820 6263 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3398b7c4d154be09432d95ec45e1029, State: Initialized, Role: FOLLOWER
I20260925 00:57:38.960242 6263 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3398b7c4d154be09432d95ec45e1029" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260925 00:57:38.960361 6263 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:38.960421 6263 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:38.960512 6263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:38.961578 6263 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3398b7c4d154be09432d95ec45e1029" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260925 00:57:38.961916 6263 leader_election.cc:302] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3398b7c4d154be09432d95ec45e1029; no voters:
I20260925 00:57:38.962211 6263 leader_election.cc:288] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:38.962314 6273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:38.962518 6273 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [term 1 LEADER]: Becoming Leader. State: Replica: e3398b7c4d154be09432d95ec45e1029, State: Running, Role: LEADER
I20260925 00:57:38.962854 6273 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3398b7c4d154be09432d95ec45e1029" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260925 00:57:38.963374 6263 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:38.964366 6275 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3398b7c4d154be09432d95ec45e1029. Latest consensus state: current_term: 1 leader_uuid: "e3398b7c4d154be09432d95ec45e1029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3398b7c4d154be09432d95ec45e1029" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } } }
I20260925 00:57:38.964475 6275 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:38.964759 6274 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3398b7c4d154be09432d95ec45e1029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3398b7c4d154be09432d95ec45e1029" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } } }
I20260925 00:57:38.964839 6274 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:38.964864 6282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:38.968055 6282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:38.974172 6282 catalog_manager.cc:1347] Generated new cluster ID: 382ae0033a9440759d1a936a22d9b478
I20260925 00:57:38.974274 6282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:38.986214 6282 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:38.986591 6282 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:38.995168 6282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3398b7c4d154be09432d95ec45e1029: Generated new TSK 0
I20260925 00:57:38.995939 6282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:38.999043 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6195
2026-09-25T00:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:39 dist-test-slave-rpk1 krb5kdc[6294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:39 dist-test-slave-rpk1 krb5kdc[6294](info): set up 2 sockets
Sep 25 00:57:39 dist-test-slave-rpk1 krb5kdc[6294](info): commencing operation
krb5kdc: starting...
W20260925 00:57:39.960937 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:40 dist-test-slave-rpk1 krb5kdc[6294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:40.086081 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38989
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:38989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:40.196205 6307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:40.196559 6307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:40.196606 6307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:40.200816 6307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:40.200908 6307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:40.200928 6307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:40.200946 6307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:40.200961 6307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:40.206583 6307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:38989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:38989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:40.207707 6307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:40.208768 6307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:40.217299 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:40.217303 6312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:40.217303 6315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:40.217339 6307 server_base.cc:1034] running on GCE node
I20260925 00:57:40.218171 6307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:40.219338 6307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:40.220511 6307 hybrid_clock.cc:648] HybridClock initialized: now 1790297860220496 us; error 29 us; skew 500 ppm
Sep 25 00:57:40 dist-test-slave-rpk1 krb5kdc[6294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:40.223213 6307 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:40.224129 6307 webserver.cc:460] Webserver started at http://127.16.202.62:33397/ using document root <none> and password file <none>
I20260925 00:57:40.224699 6307 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:40.224773 6307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:40.225003 6307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:40.227140 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "caa33f0f296c427f9974ce19ec93d88c"
format_stamp: "Formatted at 2026-09-25 00:57:40 on dist-test-slave-rpk1"
server_key: "b46577011ecab198f91dec250f8b7761"
server_key_iv: "f23d8efe75b1c9975c69313cf14f298f"
server_key_version: "encryptionkey@0"
I20260925 00:57:40.227682 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "caa33f0f296c427f9974ce19ec93d88c"
format_stamp: "Formatted at 2026-09-25 00:57:40 on dist-test-slave-rpk1"
server_key: "b46577011ecab198f91dec250f8b7761"
server_key_iv: "f23d8efe75b1c9975c69313cf14f298f"
server_key_version: "encryptionkey@0"
I20260925 00:57:40.231529 6307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:57:40.234105 6322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:40.235527 6307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:57:40.235672 6307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "caa33f0f296c427f9974ce19ec93d88c"
format_stamp: "Formatted at 2026-09-25 00:57:40 on dist-test-slave-rpk1"
server_key: "b46577011ecab198f91dec250f8b7761"
server_key_iv: "f23d8efe75b1c9975c69313cf14f298f"
server_key_version: "encryptionkey@0"
I20260925 00:57:40.235785 6307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:40.249670 6307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:40.252590 6307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:40.252799 6307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:40.259805 6307 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:38989
I20260925 00:57:40.259828 6374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:38989 every 8 connection(s)
I20260925 00:57:40.260821 6307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:40.262692 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6307
I20260925 00:57:40.262818 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:40.263084 17192 external_mini_cluster.cc:1428] Setting key 9e4f5d2b34e09bb2d337c60f25a15d4b
I20260925 00:57:40.263834 6375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:40.270931 6375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c: Bootstrap starting.
Sep 25 00:57:40 dist-test-slave-rpk1 krb5kdc[6294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:40.273546 6375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:40.274466 6375 log.cc:824] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:40.276753 6375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c: No bootstrap required, opened a new log
I20260925 00:57:40.278580 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:40.264517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59799 (local address 127.16.202.62:38989)
0925 00:57:40.264993 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:40.265012 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:40.265073 (+ 61us) server_negotiation.cc:407] Connection header received
0925 00:57:40.265919 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:40.265944 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:40.266404 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:40.266822 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:40.267546 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:40.269039 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:40.269942 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:40.270628 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:40.270800 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:40.273579 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:40.273586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:40.273605 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:40.273642 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:40.275364 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:40.276008 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:40.276014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:40.276019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:40.276140 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:40.276487 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:40.276490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:40.276491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:40.276975 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:40.277138 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:40.277528 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 00:57:40.277775 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":162,"threads_started":1}
I20260925 00:57:40.280483 6375 raft_consensus.cc:348] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa33f0f296c427f9974ce19ec93d88c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38989 } }
I20260925 00:57:40.280752 6375 raft_consensus.cc:374] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:40.280857 6375 raft_consensus.cc:729] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa33f0f296c427f9974ce19ec93d88c, State: Initialized, Role: FOLLOWER
I20260925 00:57:40.281317 6375 consensus_queue.cc:260] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa33f0f296c427f9974ce19ec93d88c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38989 } }
I20260925 00:57:40.281474 6375 raft_consensus.cc:388] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:40.281536 6375 raft_consensus.cc:482] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:40.281632 6375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:40.281862 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:40.282832 6375 raft_consensus.cc:504] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa33f0f296c427f9974ce19ec93d88c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38989 } }
I20260925 00:57:40.283217 6375 leader_election.cc:302] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caa33f0f296c427f9974ce19ec93d88c; no voters:
I20260925 00:57:40.283596 6375 leader_election.cc:288] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:40.283708 6384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:40.283958 6384 raft_consensus.cc:686] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [term 1 LEADER]: Becoming Leader. State: Replica: caa33f0f296c427f9974ce19ec93d88c, State: Running, Role: LEADER
I20260925 00:57:40.284531 6384 consensus_queue.cc:237] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa33f0f296c427f9974ce19ec93d88c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38989 } }
I20260925 00:57:40.284950 6375 sys_catalog.cc:564] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:40.286413 6386 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caa33f0f296c427f9974ce19ec93d88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa33f0f296c427f9974ce19ec93d88c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38989 } } }
I20260925 00:57:40.286444 6385 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader caa33f0f296c427f9974ce19ec93d88c. Latest consensus state: current_term: 1 leader_uuid: "caa33f0f296c427f9974ce19ec93d88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa33f0f296c427f9974ce19ec93d88c" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 38989 } } }
I20260925 00:57:40.286643 6385 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:40.286641 6386 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c [sys.catalog]: This master's current role is: LEADER
W20260925 00:57:40.290143 6400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:40.290234 6400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:40.290347 6399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:40.291545 6399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:40.294487 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:40.284098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40951 (local address 127.16.202.62:38989)
0925 00:57:40.284216 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:40.284220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:40.284455 (+ 235us) server_negotiation.cc:407] Connection header received
0925 00:57:40.284618 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:40.284621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:40.284668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:40.284776 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:40.285768 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:40.286523 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:40.287646 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:40.287962 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:40.288064 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:40.289704 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:40.289708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:40.289711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:40.289736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:40.291277 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:40.292326 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:40.292329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:40.292341 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:40.292442 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:40.292912 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:40.292916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:40.292917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:40.293241 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:40.293393 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:40.294251 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 00:57:40.294362 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 00:57:40.299582 6399 catalog_manager.cc:1347] Generated new cluster ID: f5f479efda48419a9be02934a18b3596
I20260925 00:57:40.299682 6399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:40.311199 6399 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:40.312541 6399 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:40.318065 6399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caa33f0f296c427f9974ce19ec93d88c: Generated new TSK 0
I20260925 00:57:40.318993 6399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:40.327688 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6307
W20260925 00:57:40.332667 6292 connection.cc:535] client connection to 127.16.202.62:38989 recv error: Network error: failed to read from TLS socket (remote: 127.16.202.62:38989): Connection reset by peer (error 104)
2026-09-25T00:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:40 dist-test-slave-rpk1 krb5kdc[6406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:40 dist-test-slave-rpk1 krb5kdc[6406](info): set up 2 sockets
Sep 25 00:57:40 dist-test-slave-rpk1 krb5kdc[6406](info): commencing operation
krb5kdc: starting...
W20260925 00:57:41.284936 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:41 dist-test-slave-rpk1 krb5kdc[6406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:41.405860 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45829
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:45829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:41.521448 6419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:41.521728 6419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:41.521771 6419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:41.525919 6419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:41.526005 6419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:41.526023 6419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:41.526039 6419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:41.526053 6419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:41.531693 6419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:45829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:45829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:41.532859 6419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:41.533900 6419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:41.542651 6427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:41.542675 6425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:41.542660 6424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:41.542996 6419 server_base.cc:1034] running on GCE node
I20260925 00:57:41.543684 6419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:41.544922 6419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:41.546087 6419 hybrid_clock.cc:648] HybridClock initialized: now 1790297861546061 us; error 39 us; skew 500 ppm
Sep 25 00:57:41 dist-test-slave-rpk1 krb5kdc[6406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297861, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:41.549129 6419 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:41.550099 6419 webserver.cc:460] Webserver started at http://127.16.202.62:40573/ using document root <none> and password file <none>
I20260925 00:57:41.550702 6419 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:41.550874 6419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:41.551147 6419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:41.553869 6419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "6e9cc52fd37842dca289a52ee15491b3"
format_stamp: "Formatted at 2026-09-25 00:57:41 on dist-test-slave-rpk1"
server_key: "8a448e0a9ee6d6a8c481311dab977009"
server_key_iv: "4775a49500edec5e71472f195027e447"
server_key_version: "encryptionkey@0"
I20260925 00:57:41.554522 6419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "6e9cc52fd37842dca289a52ee15491b3"
format_stamp: "Formatted at 2026-09-25 00:57:41 on dist-test-slave-rpk1"
server_key: "8a448e0a9ee6d6a8c481311dab977009"
server_key_iv: "4775a49500edec5e71472f195027e447"
server_key_version: "encryptionkey@0"
I20260925 00:57:41.558761 6419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:41.561566 6435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:41.563057 6419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:41.563202 6419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "6e9cc52fd37842dca289a52ee15491b3"
format_stamp: "Formatted at 2026-09-25 00:57:41 on dist-test-slave-rpk1"
server_key: "8a448e0a9ee6d6a8c481311dab977009"
server_key_iv: "4775a49500edec5e71472f195027e447"
server_key_version: "encryptionkey@0"
I20260925 00:57:41.563321 6419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:41.612569 6419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:41.615746 6419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:41.615947 6419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:41.622668 6419 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:45829
I20260925 00:57:41.622682 6487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:45829 every 8 connection(s)
I20260925 00:57:41.623652 6419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:41.626499 6488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:41.632035 6488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3: Bootstrap starting.
I20260925 00:57:41.633162 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6419
I20260925 00:57:41.633281 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:41.633526 17192 external_mini_cluster.cc:1428] Setting key a06ea420b4ccfc82eeab1b3781bd5a23
I20260925 00:57:41.634752 6488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:41.635468 6488 log.cc:824] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:41.637583 6488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3: No bootstrap required, opened a new log
I20260925 00:57:41.641745 6488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9cc52fd37842dca289a52ee15491b3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45829 } }
I20260925 00:57:41.641989 6488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:41.642058 6488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9cc52fd37842dca289a52ee15491b3, State: Initialized, Role: FOLLOWER
I20260925 00:57:41.642485 6488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9cc52fd37842dca289a52ee15491b3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45829 } }
I20260925 00:57:41.642590 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:41.634663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38887 (local address 127.16.202.62:45829)
0925 00:57:41.635111 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:41.635127 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:41.635162 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:57:41.635859 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:41.635885 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:41.636304 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:41.636587 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:41.636968 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:41.637952 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:41.639153 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:41.639663 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:41.639782 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:41.640739 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:41.640744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:41.640759 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:41.641174 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:41.641233 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:41.641256 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:41.641359 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:41.641547 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 00:57:41.641688 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260925 00:57:41.642613 6488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:41.642704 6488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:41.642803 6488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:41.643795 6488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9cc52fd37842dca289a52ee15491b3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45829 } }
I20260925 00:57:41.644171 6488 leader_election.cc:302] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e9cc52fd37842dca289a52ee15491b3; no voters:
I20260925 00:57:41.644548 6488 leader_election.cc:288] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:41.644651 6493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:41.644928 6493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6e9cc52fd37842dca289a52ee15491b3, State: Running, Role: LEADER
I20260925 00:57:41.645363 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9cc52fd37842dca289a52ee15491b3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45829 } }
I20260925 00:57:41.645788 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:41.645974 6488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:41.647164 6494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e9cc52fd37842dca289a52ee15491b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9cc52fd37842dca289a52ee15491b3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45829 } } }
I20260925 00:57:41.647311 6494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:41.647622 6495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e9cc52fd37842dca289a52ee15491b3. Latest consensus state: current_term: 1 leader_uuid: "6e9cc52fd37842dca289a52ee15491b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9cc52fd37842dca289a52ee15491b3" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 45829 } } }
I20260925 00:57:41.647713 6495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3 [sys.catalog]: This master's current role is: LEADER
W20260925 00:57:41.651114 6512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:41.651196 6512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:41.651269 6507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:41.652130 6507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:41.654520 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:41.649592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51571 (local address 127.16.202.62:45829)
0925 00:57:41.649700 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:41.649704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:41.650016 (+ 312us) server_negotiation.cc:407] Connection header received
0925 00:57:41.650147 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:41.650150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:41.650201 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:41.650292 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:41.650588 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:41.651383 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:41.652489 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:41.652863 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:41.652992 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:41.653934 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:41.653938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:41.653941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:41.653966 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:41.653997 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:41.654012 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:41.654102 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:41.654346 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 00:57:41.654385 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:57:41.657897 6507 catalog_manager.cc:1347] Generated new cluster ID: 4f8eb54e090d41f888dde7b203f8bf08
I20260925 00:57:41.657979 6507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:41.679253 6507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:41.680536 6507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:41.691412 6507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e9cc52fd37842dca289a52ee15491b3: Generated new TSK 0
I20260925 00:57:41.692358 6507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:41.703994 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6419
W20260925 00:57:41.708751 6404 connection.cc:535] client connection to 127.16.202.62:45829 recv error: Network error: failed to read from TLS socket (remote: 127.16.202.62:45829): Connection reset by peer (error 104)
2026-09-25T00:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:41 dist-test-slave-rpk1 krb5kdc[6519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:41 dist-test-slave-rpk1 krb5kdc[6519](info): set up 2 sockets
Sep 25 00:57:41 dist-test-slave-rpk1 krb5kdc[6519](info): commencing operation
krb5kdc: starting...
W20260925 00:57:42.676968 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:42 dist-test-slave-rpk1 krb5kdc[6519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:42.804466 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:37613
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:37613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:42.915073 6532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:42.915386 6532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:42.915447 6532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:42.919660 6532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:42.919744 6532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:42.919770 6532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:42.919793 6532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:42.919822 6532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:42.925431 6532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:37613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:42.926522 6532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:42.927536 6532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:57:42.936296 6532 server_base.cc:1034] running on GCE node
W20260925 00:57:42.936249 6540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:42.936255 6537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:42.936250 6538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:42.937150 6532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:42.938269 6532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:42.939435 6532 hybrid_clock.cc:648] HybridClock initialized: now 1790297862939421 us; error 33 us; skew 500 ppm
Sep 25 00:57:42 dist-test-slave-rpk1 krb5kdc[6519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297862, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:42.942216 6532 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:42.943112 6532 webserver.cc:460] Webserver started at http://127.16.202.62:35825/ using document root <none> and password file <none>
I20260925 00:57:42.943624 6532 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:42.943681 6532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:42.943845 6532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:42.945958 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "36c8d2743325404d861cb74e38a6937d"
format_stamp: "Formatted at 2026-09-25 00:57:42 on dist-test-slave-rpk1"
server_key: "2530e77e9cc8cac9e7c90da05d6058fb"
server_key_iv: "044deab4884ccc5a0409c5e851f6b1ca"
server_key_version: "encryptionkey@0"
I20260925 00:57:42.946506 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "36c8d2743325404d861cb74e38a6937d"
format_stamp: "Formatted at 2026-09-25 00:57:42 on dist-test-slave-rpk1"
server_key: "2530e77e9cc8cac9e7c90da05d6058fb"
server_key_iv: "044deab4884ccc5a0409c5e851f6b1ca"
server_key_version: "encryptionkey@0"
I20260925 00:57:42.950413 6532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:42.953318 6547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:42.955094 6532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:42.955245 6532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "36c8d2743325404d861cb74e38a6937d"
format_stamp: "Formatted at 2026-09-25 00:57:42 on dist-test-slave-rpk1"
server_key: "2530e77e9cc8cac9e7c90da05d6058fb"
server_key_iv: "044deab4884ccc5a0409c5e851f6b1ca"
server_key_version: "encryptionkey@0"
I20260925 00:57:42.955369 6532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:42.960268 6532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:42.963551 6532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:42.963737 6532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:42.970716 6532 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:37613
I20260925 00:57:42.970743 6599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:37613 every 8 connection(s)
I20260925 00:57:42.971787 6532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:42.974776 6600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:42.981269 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6532
I20260925 00:57:42.981420 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:42.981457 6600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d: Bootstrap starting.
I20260925 00:57:42.981737 17192 external_mini_cluster.cc:1428] Setting key 0f1acd54b6e2e0e3cde3278a774a72d1
I20260925 00:57:42.984352 6600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:42.985229 6600 log.cc:824] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:42.987838 6600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d: No bootstrap required, opened a new log
I20260925 00:57:42.987838 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:42.983242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56931 (local address 127.16.202.62:37613)
0925 00:57:42.983740 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:42.983755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:42.983795 (+ 40us) server_negotiation.cc:407] Connection header received
0925 00:57:42.984528 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:42.984550 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:42.985027 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:42.985384 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:42.986265 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:42.986271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:42.986289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:42.986302 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:42.986376 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:42.986467 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:42.986590 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:42.986715 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 00:57:42.986875 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":132,"threads_started":1}
I20260925 00:57:42.990921 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:42.992063 6600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c8d2743325404d861cb74e38a6937d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37613 } }
I20260925 00:57:42.992362 6600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:42.992426 6600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c8d2743325404d861cb74e38a6937d, State: Initialized, Role: FOLLOWER
I20260925 00:57:42.992898 6600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c8d2743325404d861cb74e38a6937d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37613 } }
I20260925 00:57:42.993054 6600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:42.993180 6600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:42.993296 6600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:42.994554 6600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c8d2743325404d861cb74e38a6937d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37613 } }
I20260925 00:57:42.994699 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:42.993044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49121 (local address 127.16.202.62:37613)
0925 00:57:42.993173 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:42.993177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:42.993306 (+ 129us) server_negotiation.cc:407] Connection header received
0925 00:57:42.993465 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:42.993468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:42.993512 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:42.993594 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:42.994300 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:42.994304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:42.994306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:42.994318 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:42.994353 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:42.994367 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:42.994471 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:42.994544 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 00:57:42.994583 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:57:42.995042 6600 leader_election.cc:302] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c8d2743325404d861cb74e38a6937d; no voters:
I20260925 00:57:42.995410 6600 leader_election.cc:288] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:42.995498 6610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:42.995713 6610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [term 1 LEADER]: Becoming Leader. State: Replica: 36c8d2743325404d861cb74e38a6937d, State: Running, Role: LEADER
I20260925 00:57:42.996145 6610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c8d2743325404d861cb74e38a6937d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37613 } }
I20260925 00:57:42.996757 6600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:42.997761 6611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c8d2743325404d861cb74e38a6937d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c8d2743325404d861cb74e38a6937d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37613 } } }
I20260925 00:57:42.997895 6611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:42.998212 6612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c8d2743325404d861cb74e38a6937d. Latest consensus state: current_term: 1 leader_uuid: "36c8d2743325404d861cb74e38a6937d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c8d2743325404d861cb74e38a6937d" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 37613 } } }
I20260925 00:57:42.998281 6619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:42.998363 6612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:43.001778 6619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:43.008143 6619 catalog_manager.cc:1347] Generated new cluster ID: 2ce2d0dff7874217adb04260ed3808d7
I20260925 00:57:43.008236 6619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:43.028720 6619 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:43.029070 6619 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:43.038298 6619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36c8d2743325404d861cb74e38a6937d: Generated new TSK 0
I20260925 00:57:43.039031 6619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:43.054318 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6532
2026-09-25T00:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:43 dist-test-slave-rpk1 krb5kdc[6631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:43 dist-test-slave-rpk1 krb5kdc[6631](info): set up 2 sockets
Sep 25 00:57:43 dist-test-slave-rpk1 krb5kdc[6631](info): commencing operation
krb5kdc: starting...
W20260925 00:57:44.016851 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:44 dist-test-slave-rpk1 krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:44.138052 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33323
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:33323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:44.250416 6644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:44.250726 6644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:44.250769 6644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:44.254956 6644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:44.255029 6644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:44.255048 6644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:44.255064 6644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:44.255079 6644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:44.260720 6644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:33323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:33323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:44.261901 6644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:44.262969 6644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:44.272207 6650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:44.272199 6652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:44.272197 6649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:44.272651 6644 server_base.cc:1034] running on GCE node
I20260925 00:57:44.273281 6644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:44.274677 6644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:44.275867 6644 hybrid_clock.cc:648] HybridClock initialized: now 1790297864275834 us; error 52 us; skew 500 ppm
Sep 25 00:57:44 dist-test-slave-rpk1 krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297864, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:44.282717 6644 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:44.283720 6644 webserver.cc:460] Webserver started at http://127.16.202.62:45953/ using document root <none> and password file <none>
I20260925 00:57:44.284297 6644 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:44.284364 6644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:44.284546 6644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:44.286669 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "b84679ab756e47898c51c557997663aa"
format_stamp: "Formatted at 2026-09-25 00:57:44 on dist-test-slave-rpk1"
server_key: "7d3385b5a3e320ddaa8f7e6942a6f7c5"
server_key_iv: "7d8bd52d8cd49f3d14d99cf69d6f266b"
server_key_version: "encryptionkey@0"
I20260925 00:57:44.287189 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "b84679ab756e47898c51c557997663aa"
format_stamp: "Formatted at 2026-09-25 00:57:44 on dist-test-slave-rpk1"
server_key: "7d3385b5a3e320ddaa8f7e6942a6f7c5"
server_key_iv: "7d8bd52d8cd49f3d14d99cf69d6f266b"
server_key_version: "encryptionkey@0"
I20260925 00:57:44.290979 6644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:44.293792 6659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:44.295269 6644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:57:44.295389 6644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "b84679ab756e47898c51c557997663aa"
format_stamp: "Formatted at 2026-09-25 00:57:44 on dist-test-slave-rpk1"
server_key: "7d3385b5a3e320ddaa8f7e6942a6f7c5"
server_key_iv: "7d8bd52d8cd49f3d14d99cf69d6f266b"
server_key_version: "encryptionkey@0"
I20260925 00:57:44.295504 6644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:44.355512 6644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:44.358631 6644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:44.358791 6644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:44.365687 6644 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:33323
I20260925 00:57:44.365762 6711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:33323 every 8 connection(s)
I20260925 00:57:44.366621 6644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:44.369397 6712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:44.375311 6712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa: Bootstrap starting.
I20260925 00:57:44.375876 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6644
I20260925 00:57:44.375991 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:44.376251 17192 external_mini_cluster.cc:1428] Setting key 5719af9f89c90af780a55443688cddef
I20260925 00:57:44.378078 6712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:44.378819 6712 log.cc:824] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:44.381081 6712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa: No bootstrap required, opened a new log
Sep 25 00:57:44 dist-test-slave-rpk1 krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:44.385455 6712 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84679ab756e47898c51c557997663aa" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33323 } }
I20260925 00:57:44.385690 6712 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:44.385748 6712 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84679ab756e47898c51c557997663aa, State: Initialized, Role: FOLLOWER
I20260925 00:57:44.386163 6712 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84679ab756e47898c51c557997663aa" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33323 } }
I20260925 00:57:44.386288 6712 raft_consensus.cc:388] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:44.386346 6712 raft_consensus.cc:482] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:44.386444 6712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:44.387470 6712 raft_consensus.cc:504] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84679ab756e47898c51c557997663aa" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33323 } }
I20260925 00:57:44.387809 6712 leader_election.cc:302] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84679ab756e47898c51c557997663aa; no voters:
I20260925 00:57:44.388108 6712 leader_election.cc:288] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:44.388195 6717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:44.388360 6717 raft_consensus.cc:686] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [term 1 LEADER]: Becoming Leader. State: Replica: b84679ab756e47898c51c557997663aa, State: Running, Role: LEADER
I20260925 00:57:44.388738 6717 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84679ab756e47898c51c557997663aa" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33323 } }
I20260925 00:57:44.389452 6712 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:44.390707 6719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84679ab756e47898c51c557997663aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84679ab756e47898c51c557997663aa" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33323 } } }
I20260925 00:57:44.390741 6718 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84679ab756e47898c51c557997663aa. Latest consensus state: current_term: 1 leader_uuid: "b84679ab756e47898c51c557997663aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84679ab756e47898c51c557997663aa" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 33323 } } }
I20260925 00:57:44.390846 6718 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:44.390832 6719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:44.391160 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:44.377699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42589 (local address 127.16.202.62:33323)
0925 00:57:44.378146 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:44.378162 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:44.378203 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:57:44.378877 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:44.378896 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:44.379348 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:44.379713 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:44.380585 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:44.381670 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:44.382821 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:44.383311 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:44.383426 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:44.386186 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:44.386191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:44.386206 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:44.386234 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:44.387891 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:44.388365 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:44.388372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:44.388379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:44.388525 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:44.388858 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:44.388861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:44.388863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:44.389280 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:44.389477 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:44.389751 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 00:57:44.390048 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260925 00:57:44.394289 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 00:57:44.394593 6732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:44.394671 6732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:44.394747 6728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:44.395601 6728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:44.401577 6728 catalog_manager.cc:1347] Generated new cluster ID: e064e45e36b74e6e8e9412e7b3b21446
I20260925 00:57:44.401639 6728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:44.405308 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:44.396213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45517 (local address 127.16.202.62:33323)
0925 00:57:44.396321 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:44.396325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:44.396539 (+ 214us) server_negotiation.cc:407] Connection header received
0925 00:57:44.396648 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:44.396651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:44.396718 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:44.396786 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:44.397500 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:44.398233 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:44.399780 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:44.400102 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:44.400195 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:44.401622 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:44.401626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:44.401629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:44.401654 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:44.403247 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:44.403698 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:44.403701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:44.403703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:44.403778 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:44.404046 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:44.404049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:44.404050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:44.404237 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:44.404334 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:44.405043 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 00:57:44.405152 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:57:44.408300 6728 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:44.409286 6728 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:44.431979 6728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b84679ab756e47898c51c557997663aa: Generated new TSK 0
I20260925 00:57:44.432806 6728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:44.440864 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6644
2026-09-25T00:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:44 dist-test-slave-rpk1 krb5kdc[6743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:44 dist-test-slave-rpk1 krb5kdc[6743](info): set up 2 sockets
Sep 25 00:57:44 dist-test-slave-rpk1 krb5kdc[6743](info): commencing operation
krb5kdc: starting...
W20260925 00:57:45.396960 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:45 dist-test-slave-rpk1 krb5kdc[6743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:45.523487 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39859
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:45.644404 6756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:45.644766 6756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:45.644832 6756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:45.649063 6756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:45.649142 6756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:45.649168 6756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:45.649192 6756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:45.649221 6756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:45.654807 6756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:45.655965 6756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:45.657045 6756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:45.665925 6762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:45.665980 6761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:45.665904 6764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:45.666913 6756 server_base.cc:1034] running on GCE node
I20260925 00:57:45.667608 6756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:45.668879 6756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:45.670055 6756 hybrid_clock.cc:648] HybridClock initialized: now 1790297865670011 us; error 67 us; skew 500 ppm
Sep 25 00:57:45 dist-test-slave-rpk1 krb5kdc[6743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297865, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:45.673281 6756 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:45.674353 6756 webserver.cc:460] Webserver started at http://127.16.202.62:43835/ using document root <none> and password file <none>
I20260925 00:57:45.674935 6756 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:45.675006 6756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:45.675206 6756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:45.677416 6756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "7854b88aed9141c289a99e19d2419704"
format_stamp: "Formatted at 2026-09-25 00:57:45 on dist-test-slave-rpk1"
server_key: "b7eb108bca3edd181416b3b158ee9667"
server_key_iv: "602c1fe09944b271f43fa91f52b59abf"
server_key_version: "encryptionkey@0"
I20260925 00:57:45.678009 6756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "7854b88aed9141c289a99e19d2419704"
format_stamp: "Formatted at 2026-09-25 00:57:45 on dist-test-slave-rpk1"
server_key: "b7eb108bca3edd181416b3b158ee9667"
server_key_iv: "602c1fe09944b271f43fa91f52b59abf"
server_key_version: "encryptionkey@0"
I20260925 00:57:45.682060 6756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 00:57:45.684909 6771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:45.686357 6756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:57:45.686501 6756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "7854b88aed9141c289a99e19d2419704"
format_stamp: "Formatted at 2026-09-25 00:57:45 on dist-test-slave-rpk1"
server_key: "b7eb108bca3edd181416b3b158ee9667"
server_key_iv: "602c1fe09944b271f43fa91f52b59abf"
server_key_version: "encryptionkey@0"
I20260925 00:57:45.686631 6756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:45.701530 6756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:45.705148 6756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:45.705360 6756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:45.712641 6756 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39859
I20260925 00:57:45.712694 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39859 every 8 connection(s)
I20260925 00:57:45.713687 6756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:45.716572 6824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:45.720342 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6756
I20260925 00:57:45.720476 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:45.720791 17192 external_mini_cluster.cc:1428] Setting key 9dc13aa1e014f7323e3c999b72c4bc4d
I20260925 00:57:45.722883 6824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704: Bootstrap starting.
I20260925 00:57:45.725179 6824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:45.726019 6824 log.cc:824] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:45.728283 6824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704: No bootstrap required, opened a new log
I20260925 00:57:45.729835 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:45.722271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40963 (local address 127.16.202.62:39859)
0925 00:57:45.722687 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:45.722703 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:45.722737 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:57:45.723394 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:45.723412 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:45.723750 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:45.724031 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:45.724355 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:45.725395 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:45.726538 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:45.727121 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:45.727246 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:45.728069 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:45.728075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:45.728087 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:45.728499 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:45.728558 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:45.728582 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:45.728662 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:45.728833 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 00:57:45.729041 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260925 00:57:45.732051 6824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854b88aed9141c289a99e19d2419704" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39859 } }
I20260925 00:57:45.732323 6824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:45.732383 6824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7854b88aed9141c289a99e19d2419704, State: Initialized, Role: FOLLOWER
I20260925 00:57:45.732805 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:45.732869 6824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854b88aed9141c289a99e19d2419704" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39859 } }
I20260925 00:57:45.733036 6824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:45.733095 6824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:45.733194 6824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:45.734575 6824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854b88aed9141c289a99e19d2419704" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39859 } }
I20260925 00:57:45.735001 6824 leader_election.cc:302] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7854b88aed9141c289a99e19d2419704; no voters:
I20260925 00:57:45.735364 6824 leader_election.cc:288] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:45.735422 6834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:45.735620 6834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [term 1 LEADER]: Becoming Leader. State: Replica: 7854b88aed9141c289a99e19d2419704, State: Running, Role: LEADER
I20260925 00:57:45.736032 6834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854b88aed9141c289a99e19d2419704" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39859 } }
I20260925 00:57:45.736811 6824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:45.737747 6836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7854b88aed9141c289a99e19d2419704. Latest consensus state: current_term: 1 leader_uuid: "7854b88aed9141c289a99e19d2419704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854b88aed9141c289a99e19d2419704" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39859 } } }
I20260925 00:57:45.737898 6836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:45.738318 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7854b88aed9141c289a99e19d2419704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854b88aed9141c289a99e19d2419704" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39859 } } }
I20260925 00:57:45.738420 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:45.738804 6843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:45.740113 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:45.735239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54559 (local address 127.16.202.62:39859)
0925 00:57:45.735362 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:45.735368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:45.735648 (+ 280us) server_negotiation.cc:407] Connection header received
0925 00:57:45.735848 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:45.735852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:45.735901 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:45.735950 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:45.736366 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:45.736974 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:45.738146 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:45.738466 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:45.738541 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:45.739658 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:45.739662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:45.739665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:45.739680 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:45.739711 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:45.739725 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:45.739790 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:45.739912 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 00:57:45.739976 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 00:57:45.742321 6843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:45.748855 6843 catalog_manager.cc:1347] Generated new cluster ID: aba25fb2e73d4670a1ecc18a79e13e58
I20260925 00:57:45.748957 6843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:45.778003 6843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:45.779552 6843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:45.785487 6843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7854b88aed9141c289a99e19d2419704: Generated new TSK 0
I20260925 00:57:45.786329 6843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:45.799252 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6756
2026-09-25T00:57:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:45 dist-test-slave-rpk1 krb5kdc[6855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:45 dist-test-slave-rpk1 krb5kdc[6855](info): set up 2 sockets
Sep 25 00:57:45 dist-test-slave-rpk1 krb5kdc[6855](info): commencing operation
krb5kdc: starting...
W20260925 00:57:46.772971 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:46 dist-test-slave-rpk1 krb5kdc[6855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:46.895973 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:43417
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:43417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:47.008535 6868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:47.008862 6868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:47.008910 6868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:47.013091 6868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:47.013180 6868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:47.013199 6868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:47.013216 6868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:47.013229 6868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:47.018822 6868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:43417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:43417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:47.020012 6868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:47.021064 6868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:47.029717 6874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:47.029733 6873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:47.029735 6876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:47.030335 6868 server_base.cc:1034] running on GCE node
I20260925 00:57:47.031081 6868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:47.032335 6868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:47.033519 6868 hybrid_clock.cc:648] HybridClock initialized: now 1790297867033493 us; error 45 us; skew 500 ppm
Sep 25 00:57:47 dist-test-slave-rpk1 krb5kdc[6855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297867, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:47.036336 6868 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:47.037374 6868 webserver.cc:460] Webserver started at http://127.16.202.62:35927/ using document root <none> and password file <none>
I20260925 00:57:47.037949 6868 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:47.038022 6868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:47.038224 6868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:47.040397 6868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "0b5560f6fa06469c95bcf69fb70e6f61"
format_stamp: "Formatted at 2026-09-25 00:57:47 on dist-test-slave-rpk1"
server_key: "dc145c8425583063072de991dd6e45e2"
server_key_iv: "9a742df900aa6c9502b382ffa1afb67e"
server_key_version: "encryptionkey@0"
I20260925 00:57:47.041101 6868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "0b5560f6fa06469c95bcf69fb70e6f61"
format_stamp: "Formatted at 2026-09-25 00:57:47 on dist-test-slave-rpk1"
server_key: "dc145c8425583063072de991dd6e45e2"
server_key_iv: "9a742df900aa6c9502b382ffa1afb67e"
server_key_version: "encryptionkey@0"
I20260925 00:57:47.046991 6868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 00:57:47.050426 6883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:47.051957 6868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:57:47.052119 6868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "0b5560f6fa06469c95bcf69fb70e6f61"
format_stamp: "Formatted at 2026-09-25 00:57:47 on dist-test-slave-rpk1"
server_key: "dc145c8425583063072de991dd6e45e2"
server_key_iv: "9a742df900aa6c9502b382ffa1afb67e"
server_key_version: "encryptionkey@0"
I20260925 00:57:47.052258 6868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:47.057144 6868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:47.060840 6868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:47.061050 6868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:47.068022 6935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:43417 every 8 connection(s)
I20260925 00:57:47.068146 6868 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:43417
I20260925 00:57:47.069226 6868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:47.072430 6936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:47.074152 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6868
I20260925 00:57:47.074290 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:47.074545 17192 external_mini_cluster.cc:1428] Setting key f63e76ae0f721a492d07c3bbf7446fc8
I20260925 00:57:47.078907 6936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61: Bootstrap starting.
I20260925 00:57:47.079919 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:47.075978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40047 (local address 127.16.202.62:43417)
0925 00:57:47.076502 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:47.076518 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:47.076555 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:57:47.077245 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:47.077263 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:47.077582 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:47.077908 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:47.078690 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:47.078694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:47.078707 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:47.078718 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:47.078774 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:47.078794 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:47.078926 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:47.078997 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 00:57:47.079164 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260925 00:57:47.081365 6936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:47.082208 6936 log.cc:824] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:47.083112 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:47.084884 6936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61: No bootstrap required, opened a new log
I20260925 00:57:47.087421 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:47.085670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40201 (local address 127.16.202.62:43417)
0925 00:57:47.085809 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:47.085814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:47.086014 (+ 200us) server_negotiation.cc:407] Connection header received
0925 00:57:47.086113 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:47.086116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:47.086152 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:47.086236 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:47.086964 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:47.086968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:47.086971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:57:47.086984 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:47.087014 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:57:47.087027 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:47.087121 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:47.087243 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 00:57:47.087283 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:57:47.089496 6936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 43417 } }
I20260925 00:57:47.089740 6936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:57:47.089773 6936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5560f6fa06469c95bcf69fb70e6f61, State: Initialized, Role: FOLLOWER
I20260925 00:57:47.090174 6936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 43417 } }
I20260925 00:57:47.090286 6936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:57:47.090334 6936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:57:47.090407 6936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:47.091327 6936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 43417 } }
I20260925 00:57:47.091658 6936 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5560f6fa06469c95bcf69fb70e6f61; no voters:
I20260925 00:57:47.091948 6936 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:57:47.092162 6946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:47.092372 6946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5560f6fa06469c95bcf69fb70e6f61, State: Running, Role: LEADER
I20260925 00:57:47.092801 6946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 43417 } }
I20260925 00:57:47.093669 6936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:47.094666 6948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5560f6fa06469c95bcf69fb70e6f61. Latest consensus state: current_term: 1 leader_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 43417 } } }
I20260925 00:57:47.094666 6947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5560f6fa06469c95bcf69fb70e6f61" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 43417 } } }
I20260925 00:57:47.094826 6947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:47.094826 6948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:47.095386 6955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:47.098642 6955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:47.105712 6955 catalog_manager.cc:1347] Generated new cluster ID: 21d9aa82b2e749828fa7d46749fa2d4d
I20260925 00:57:47.105857 6955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:47.116063 6955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:47.116408 6955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:47.128024 6955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5560f6fa06469c95bcf69fb70e6f61: Generated new TSK 0
I20260925 00:57:47.129041 6955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:57:47.142354 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6868
2026-09-25T00:57:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1343 ms)
[----------] 18 tests from AuthTokenIssuingTest (24255 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:47 dist-test-slave-rpk1 krb5kdc[6967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:47 dist-test-slave-rpk1 krb5kdc[6967](info): set up 2 sockets
Sep 25 00:57:47 dist-test-slave-rpk1 krb5kdc[6967](info): commencing operation
krb5kdc: starting...
W20260925 00:57:48.104931 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:48.228379 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:32937
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:32937,127.16.202.61:38379,127.16.202.60:43135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:48.341099 6980 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:57:48.341425 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:48.341471 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:48.341537 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:48.345803 6980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:48.345876 6980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:48.345902 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:48.345924 6980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:48.345950 6980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:48.351452 6980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:32937,127.16.202.61:38379,127.16.202.60:43135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:48.352582 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:48.353662 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:48.362979 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:48.362982 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:48.362982 6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:48.363042 6980 server_base.cc:1034] running on GCE node
I20260925 00:57:48.363849 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:48.365149 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:48.366312 6980 hybrid_clock.cc:648] HybridClock initialized: now 1790297868366290 us; error 36 us; skew 500 ppm
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:48.369092 6980 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:48.370067 6980 webserver.cc:460] Webserver started at http://127.16.202.62:46383/ using document root <none> and password file <none>
I20260925 00:57:48.370656 6980 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:48.370723 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:48.370923 6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:48.373134 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "5c831f55f840415d854e8af0f14893bb"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "cf9bf4f42e1cb9ec22337b1c48bea92e"
server_key_iv: "2dff1fe28da4d9f7ecebca3be986101f"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.373741 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "5c831f55f840415d854e8af0f14893bb"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "cf9bf4f42e1cb9ec22337b1c48bea92e"
server_key_iv: "2dff1fe28da4d9f7ecebca3be986101f"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.378101 6980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:48.380667 6995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:48.382243 6980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:48.382391 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "5c831f55f840415d854e8af0f14893bb"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "cf9bf4f42e1cb9ec22337b1c48bea92e"
server_key_iv: "2dff1fe28da4d9f7ecebca3be986101f"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.382529 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:48.401626 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:48.404652 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:48.404848 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:48.411530 6980 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:32937
I20260925 00:57:48.411850 7047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:32937 every 8 connection(s)
I20260925 00:57:48.412495 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:48.415300 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 6980
I20260925 00:57:48.415403 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:48.415436 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:48.415711 17192 external_mini_cluster.cc:1428] Setting key e5b1dede043693c60819513662948304
I20260925 00:57:48.419914 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.202.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.61@KRBTEST.COM" created.
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:48.435698 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.422527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:32937 (local address 127.0.0.1:58688)
0925 00:57:48.423095 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.423138 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.424084 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.424754 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.424759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.425201 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.425452 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.425466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.426571 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.426573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.427325 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.427335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.428050 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.428055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.428149 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.428695 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.428718 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.430332 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.432448 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.432591 (+ 143us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.432605 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.432934 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.433277 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.433280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.433282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.433396 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.433997 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.434005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.434138 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.434283 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.434700 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":87,"thread_start_us":176,"threads_started":1}
I20260925 00:57:48.435699 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.423561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.16.202.62:32937)
0925 00:57:48.423864 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.423876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.423921 (+ 45us) server_negotiation.cc:407] Connection header received
0925 00:57:48.424374 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.424378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.424561 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.424748 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.425586 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.426446 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.427435 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.427866 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.428039 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.430481 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.430489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.430504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.430529 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.432305 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.433062 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.433067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.433068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.433189 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.433487 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.433489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.433490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.433888 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.434022 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.434387 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.434611 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260925 00:57:48.438663 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:48.439457 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.439165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45320)
0925 00:57:48.439294 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.439377 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 00:57:48.439661 6997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
I20260925 00:57:48.440723 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.440447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45328)
0925 00:57:48.440568 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.440622 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 00:57:48.440865 7048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:38379: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab.
Entry for principal kudu/127.16.202.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab.
I20260925 00:57:48.443492 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:38379
--webserver_interface=127.16.202.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab
--principal=kudu/127.16.202.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:32937,127.16.202.61:38379,127.16.202.60:43135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:57:48.490154 7048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } attempt: 1
I20260925 00:57:48.491210 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.490842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45340)
0925 00:57:48.491023 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.491112 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 00:57:48.491978 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.491735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45352)
0925 00:57:48.491847 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.491910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:57:48.492148 7048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:38379: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
W20260925 00:57:48.557365 7055 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:57:48.557731 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:48.557776 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:48.557843 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 00:57:48.560379 7048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } attempt: 2
I20260925 00:57:48.561292 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.560957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45356)
0925 00:57:48.561133 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.561198 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 00:57:48.562036 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.561813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45358)
0925 00:57:48.561931 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.561975 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:57:48.562167 7048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:38379: Network error: Client connection negotiation failed: client connection to 127.16.202.61:38379: connect: Connection refused (error 111)
W20260925 00:57:48.562518 7055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:48.562574 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:48.562594 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:48.562624 7055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:48.562654 7055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:48.568431 7055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:32937,127.16.202.61:38379,127.16.202.60:43135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab
--principal=kudu/127.16.202.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:38379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.202.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:48.569679 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:48.570680 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:48.580073 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:48.580073 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:48.580158 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:48.580261 7055 server_base.cc:1034] running on GCE node
I20260925 00:57:48.580935 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:48.582058 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:48.583209 7055 hybrid_clock.cc:648] HybridClock initialized: now 1790297868583192 us; error 34 us; skew 500 ppm
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:48.586045 7055 init.cc:377] Logged in from keytab as kudu/127.16.202.61@KRBTEST.COM (short username kudu)
I20260925 00:57:48.586933 7055 webserver.cc:460] Webserver started at http://127.16.202.61:40059/ using document root <none> and password file <none>
I20260925 00:57:48.587538 7055 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:48.587611 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:48.587813 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:48.589969 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data/instance:
uuid: "63e32b9ea7414ee38159e2745c5dd09a"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "c931ba0bdc742cbe8641a86b1a9541f0"
server_key_iv: "2b22a1d51fe24f7ed799bebb83179047"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.590574 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal/instance:
uuid: "63e32b9ea7414ee38159e2745c5dd09a"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "c931ba0bdc742cbe8641a86b1a9541f0"
server_key_iv: "2b22a1d51fe24f7ed799bebb83179047"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.594522 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:48.597447 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:48.598811 7055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:48.598932 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal
uuid: "63e32b9ea7414ee38159e2745c5dd09a"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "c931ba0bdc742cbe8641a86b1a9541f0"
server_key_iv: "2b22a1d51fe24f7ed799bebb83179047"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.599087 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:48.622862 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:48.626137 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:48.626329 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:48.634038 7055 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.61:38379
I20260925 00:57:48.634361 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.61:38379 every 8 connection(s)
I20260925 00:57:48.635033 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
I20260925 00:57:48.637954 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:48.640654 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 7055
I20260925 00:57:48.640805 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-1/wal/instance
I20260925 00:57:48.641106 17192 external_mini_cluster.cc:1428] Setting key e31b9021f65e0694ac6b824130bf6bda
I20260925 00:57:48.642913 7124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.202.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.60@KRBTEST.COM" created.
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:48.653337 7048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } attempt: 3
I20260925 00:57:48.658056 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.644891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.16.202.62:32937)
0925 00:57:48.645065 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.645071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.646023 (+ 952us) server_negotiation.cc:407] Connection header received
0925 00:57:48.647004 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.647009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.647072 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.647182 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.648621 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.649432 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.650792 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.651051 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.651142 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.654102 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.654106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.654109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.654131 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.655629 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.656369 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.656373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.656375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.656472 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.656864 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.656867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.656869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.657058 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.657143 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.657720 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.657874 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:57:48.659334 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.645345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:32937 (local address 127.0.0.1:58692)
0925 00:57:48.645848 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.645901 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.646784 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.647387 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.647395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.647783 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.648371 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.648393 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.649596 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.649599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.650619 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.650632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.651190 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.651192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.651367 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.652084 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.652106 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.653882 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.655745 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.655752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.655770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.656271 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.656546 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.656550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.656552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.656768 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.657156 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.657163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.657429 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.657631 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.657976 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:57:48.661793 7124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:48.664259 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.653816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45360)
0925 00:57:48.654012 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.654027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.654100 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.654957 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.654969 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.655192 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.655332 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.655339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.656538 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.656543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.657400 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.657412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.658217 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.658221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.658328 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.658842 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.658855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.660349 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.662437 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.662441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.662444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.662704 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.663057 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.663060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.663061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.663149 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.663725 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.663728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.663836 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.663935 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.664090 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 00:57:48.664336 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.654152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.16.202.61:38379)
0925 00:57:48.654479 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.654490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.654511 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:57:48.654548 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.654555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.654791 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.654945 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.655462 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.656381 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.657524 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.658068 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.658553 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.660526 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.660534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.660545 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.660568 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.662316 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.662827 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.662833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.662835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.662924 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.663267 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.663270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.663271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.663623 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.663761 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.664031 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.664176 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
I20260925 00:57:48.666159 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:48.666901 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.666637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42314)
0925 00:57:48.666775 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.666830 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:57:48.667711 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.667437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42324)
0925 00:57:48.667613 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.667654 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260925 00:57:48.667835 7048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:43135: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
I20260925 00:57:48.670674 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.662365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45362)
0925 00:57:48.662531 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.662542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.662596 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.663056 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.663059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.663205 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.663312 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.663319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.664046 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.664049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.664858 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.664868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.665785 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.665789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.665875 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.666315 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.666328 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.667721 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.669382 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.669385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.669386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.669584 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.669836 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.669838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.669839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.669917 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.670245 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.670247 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.670336 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.670420 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.670559 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 00:57:48.670769 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.662570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.16.202.61:38379)
0925 00:57:48.662877 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.662887 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.662898 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:57:48.662926 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.662929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.662978 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.663053 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.663410 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.663947 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.664954 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.665682 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.665770 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.667873 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.667877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.667879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.667903 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.669281 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.669678 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.669682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.669683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.669766 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.670002 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.670004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.670005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.670181 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.670276 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.670455 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.670618 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
I20260925 00:57:48.671406 7124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab.
Entry for principal kudu/127.16.202.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab.
I20260925 00:57:48.672132 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.671869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42326)
0925 00:57:48.671996 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.672075 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:57:48.672408 7073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
I20260925 00:57:48.672453 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:43135
--webserver_interface=127.16.202.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab
--principal=kudu/127.16.202.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:32937,127.16.202.61:38379,127.16.202.60:43135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:57:48.673281 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.673064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42328)
0925 00:57:48.673187 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.673233 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 00:57:48.673454 7124 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:43135: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
I20260925 00:57:48.699031 7048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } attempt: 1
I20260925 00:57:48.699906 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.699604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42342)
0925 00:57:48.699760 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.699825 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 00:57:48.700795 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.700528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42346)
0925 00:57:48.700639 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.700732 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:57:48.700929 7048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:43135: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
I20260925 00:57:48.722764 7124 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } attempt: 1
I20260925 00:57:48.723665 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.723374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42360)
0925 00:57:48.723527 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.723588 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 00:57:48.724400 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.724173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42362)
0925 00:57:48.724259 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.724302 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 00:57:48.724519 7124 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:43135: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
I20260925 00:57:48.776095 7048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } attempt: 2
I20260925 00:57:48.776903 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.776594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42372)
0925 00:57:48.776760 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.776820 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 00:57:48.777722 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.777482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42384)
0925 00:57:48.777611 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.777660 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 00:57:48.777872 7048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:43135: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
W20260925 00:57:48.787169 7132 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:57:48.787506 7132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:48.787551 7132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:48.787608 7132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:48.791810 7132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:48.791887 7132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:48.791914 7132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:48.791937 7132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:48.791960 7132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:48.792706 7124 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } attempt: 2
I20260925 00:57:48.793529 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.793228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42392)
0925 00:57:48.793382 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.793452 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 00:57:48.794193 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.793988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42400)
0925 00:57:48.794091 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.794135 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 00:57:48.794304 7124 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:43135: Network error: Client connection negotiation failed: client connection to 127.16.202.60:43135: connect: Connection refused (error 111)
I20260925 00:57:48.797760 7132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:32937,127.16.202.61:38379,127.16.202.60:43135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab
--principal=kudu/127.16.202.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.202.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:48.798882 7132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:48.800091 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:48.809217 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:48.809259 7137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:48.809369 7132 server_base.cc:1034] running on GCE node
W20260925 00:57:48.809237 7140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:48.810150 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:48.811400 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:48.812608 7132 hybrid_clock.cc:648] HybridClock initialized: now 1790297868812581 us; error 41 us; skew 500 ppm
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:48.815735 7132 init.cc:377] Logged in from keytab as kudu/127.16.202.60@KRBTEST.COM (short username kudu)
I20260925 00:57:48.816780 7132 webserver.cc:460] Webserver started at http://127.16.202.60:44195/ using document root <none> and password file <none>
I20260925 00:57:48.817370 7132 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:48.817446 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:48.817657 7132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:48.820019 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data/instance:
uuid: "4fd6866565b04f9d85454275bf1dc007"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "43747948c3056801ce41778a96d39b06"
server_key_iv: "be2638c31de66f60f66f6839633c5123"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.820655 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal/instance:
uuid: "4fd6866565b04f9d85454275bf1dc007"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "43747948c3056801ce41778a96d39b06"
server_key_iv: "be2638c31de66f60f66f6839633c5123"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.825084 7132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:57:48.828100 7147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:48.829746 7132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:48.829933 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal
uuid: "4fd6866565b04f9d85454275bf1dc007"
format_stamp: "Formatted at 2026-09-25 00:57:48 on dist-test-slave-rpk1"
server_key: "43747948c3056801ce41778a96d39b06"
server_key_iv: "be2638c31de66f60f66f6839633c5123"
server_key_version: "encryptionkey@0"
I20260925 00:57:48.830065 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:48.842805 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:48.845980 7132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:48.846177 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:48.853219 7132 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.60:43135
I20260925 00:57:48.853518 7199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.60:43135 every 8 connection(s)
I20260925 00:57:48.854211 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
I20260925 00:57:48.857124 7200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:48.859297 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 7132
I20260925 00:57:48.859428 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297591180183-17192-0/minicluster-data/master-2/wal/instance
I20260925 00:57:48.859689 17192 external_mini_cluster.cc:1428] Setting key 695e5362e92f422be46b5da0bcf9b12c
I20260925 00:57:48.861307 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
I20260925 00:57:48.872611 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.863276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.16.202.62:32937)
0925 00:57:48.863398 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.863402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.863778 (+ 376us) server_negotiation.cc:407] Connection header received
0925 00:57:48.863894 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.863897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.863950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.864025 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.864916 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.865405 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.866403 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.866658 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.866757 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.869033 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.869038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.869041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.869070 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.870485 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.871005 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.871007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.871009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.871078 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.871385 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.871389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.871391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.871653 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.871745 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.872304 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.872430 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:57:48.873607 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.863221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:32937 (local address 127.0.0.1:58700)
0925 00:57:48.863633 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.863681 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.863803 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.864077 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.864084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.864553 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.864784 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.864800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.865526 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.865528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.866291 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.866299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.866802 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.866806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.866898 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.867430 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.867453 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.868869 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.870587 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.870594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.870606 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.870921 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.871147 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.871150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.871152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.871300 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.871743 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.871748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.872073 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.872238 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.872550 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260925 00:57:48.873607 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.861114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.16.202.60:43135)
0925 00:57:48.861559 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.861578 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.861621 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:57:48.862495 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.862517 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.862914 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.863215 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.864040 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.865309 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.866152 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.866585 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.866694 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.869440 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.869447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.869460 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.869486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.871125 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.871524 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.871529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.871530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.871592 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.871844 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.871846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.871848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.872123 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.872304 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.872530 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.872819 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260925 00:57:48.874781 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:48.877032 7048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } attempt: 3
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
I20260925 00:57:48.884287 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.875238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:38379 (local address 127.0.0.1:45368)
0925 00:57:48.875397 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.875410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.875480 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.875721 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.875724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.875896 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.876037 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.876045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.877044 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.877048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.877830 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.877839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.878402 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.878405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.878551 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.879044 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.879057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.880454 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.882372 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.882375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.882377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.882860 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.883247 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.883250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.883251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.883344 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.883780 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.883782 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.883855 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.883951 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.884118 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":23}
I20260925 00:57:48.884287 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.875363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.16.202.61:38379)
0925 00:57:48.875527 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.875531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.875544 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:57:48.875579 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.875581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.875619 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.875689 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.876143 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.876948 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.877966 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.878301 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.878390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.880613 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.880617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.880620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.880648 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.882252 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.883019 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.883024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.883026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.883145 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.883457 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.883461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.883463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.883662 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.883723 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.884011 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.884124 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:57:48.885453 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:48.885437 7124 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } attempt: 3
I20260925 00:57:48.885795 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.875889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.16.202.62:32937)
0925 00:57:48.876000 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.876005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.876019 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:57:48.876053 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.876056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.876099 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.876168 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.876850 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.877608 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.878553 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.878841 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.878935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.881978 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.881982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.881985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.882009 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.883913 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.884397 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.884401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.884403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.884486 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.885000 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.885001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.885002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.885164 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.885390 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.885533 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.885646 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:57:48.886200 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.877367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42414)
0925 00:57:48.877525 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.877537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.877629 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.877849 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.877852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.878022 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.878106 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.878111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.878900 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.878903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.879459 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.879470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.879995 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.879997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.880058 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.880489 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.880500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.882323 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.884230 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.884234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.884236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.884511 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.884812 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.884815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.884817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.884904 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.885450 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.885453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.885537 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.885607 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.885708 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 00:57:48.886678 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.877417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.16.202.60:43135)
0925 00:57:48.877508 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.877512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.877575 (+ 63us) server_negotiation.cc:407] Connection header received
0925 00:57:48.877721 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.877724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.877761 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.877817 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.878205 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.878802 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.879562 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.879898 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.880002 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.882425 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.882429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.882432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.882460 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.884116 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.884617 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.884620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.884621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.884722 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.885092 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.885094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.885096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.885299 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.886352 (+ 1053us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.886437 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.886545 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":37}
I20260925 00:57:48.890151 7048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: Bootstrap starting.
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
Sep 25 00:57:48 dist-test-slave-rpk1 krb5kdc[6967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:57:48.893198 7048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:48.894183 7048 log.cc:824] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:48.896955 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.887520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.16.202.61:38379)
0925 00:57:48.887621 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.887625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.887637 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:57:48.887670 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.887673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.887709 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.887786 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.888473 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.889208 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.890240 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.890771 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.890862 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.893557 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.893560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.893563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.893588 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.895031 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895492 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.895496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.895498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.895584 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895865 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.895868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.895870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.896083 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.896247 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.896610 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.896786 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":223,"server-negotiator.queue_time_us":45}
I20260925 00:57:48.896976 7048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: No bootstrap required, opened a new log
I20260925 00:57:48.897056 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.885886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42432)
0925 00:57:48.886967 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.886979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.887048 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.887954 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.887957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.888107 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.888179 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.888185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.889125 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.889128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.889871 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.889878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.890543 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.890546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.890619 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.891192 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.891207 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.892877 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.895666 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.895670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.895870 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.896120 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.896122 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.896124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.896198 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.896540 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.896542 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.896611 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.896743 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.896865 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021}
I20260925 00:57:48.897068 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.886031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.16.202.60:43135)
0925 00:57:48.886707 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.886718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.886730 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:57:48.886766 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.886769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.886811 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.887310 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.887412 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.888628 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.889711 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.890074 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.893169 (+ 3095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.893326 (+ 157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.893329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.893332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.893352 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.895320 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895791 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.895794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.895796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.895867 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.896136 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.896139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.896140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.896329 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.896418 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.896832 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.896916 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":115,"threads_started":1}
I20260925 00:57:48.897480 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.886383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.16.202.60:43135)
0925 00:57:48.886679 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.886687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.887311 (+ 624us) server_negotiation.cc:407] Connection header received
0925 00:57:48.887341 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.887343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.887377 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.887461 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.888282 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.889027 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.889967 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.890343 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.890443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.893025 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.893029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.893031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.893054 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.895567 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895963 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.895966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.895967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.896035 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.896278 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.896281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.896283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.896442 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.896510 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.897243 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.897348 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":71,"threads_started":1}
I20260925 00:57:48.897964 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.885772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:43135 (local address 127.0.0.1:42416)
0925 00:57:48.885886 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:48.885899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:48.885971 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:48.886947 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:48.886950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:48.887112 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:48.887211 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.887218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.888772 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.888775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.889585 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:48.889594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.890184 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.890188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:48.890288 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.890985 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:48.890998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:48.892996 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:48.895454 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.895459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.895693 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.895965 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:48.895968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:48.895970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:48.896048 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:48.896463 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:48.896465 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:48.896558 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 00:57:48.896651 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 00:57:48.897807 (+ 1156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":970}
I20260925 00:57:48.900400 7200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: Bootstrap starting.
I20260925 00:57:48.900839 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: Bootstrap starting.
I20260925 00:57:48.901387 7048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.901643 7048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c831f55f840415d854e8af0f14893bb, State: Initialized, Role: FOLLOWER
I20260925 00:57:48.901986 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:48.901971 7048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.903385 7048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:48.903421 7124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:48.903422 7200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:48.903429 7022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c831f55f840415d854e8af0f14893bb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58704
I20260925 00:57:48.903621 7022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 00:57:48.903716 7022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:48.904297 7124 log.cc:824] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:48.904299 7200 log.cc:824] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:48.905174 7022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.906193 7022 leader_election.cc:288] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [CANDIDATE]: Term 1 election: Requested vote from peers 63e32b9ea7414ee38159e2745c5dd09a (127.16.202.61:38379), 4fd6866565b04f9d85454275bf1dc007 (127.16.202.60:43135)
I20260925 00:57:48.906558 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: No bootstrap required, opened a new log
I20260925 00:57:48.906558 7200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: No bootstrap required, opened a new log
W20260925 00:57:48.906879 7001 tablet.cc:2369] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:57:48.910629 7124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.910878 7124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e32b9ea7414ee38159e2745c5dd09a, State: Initialized, Role: FOLLOWER
I20260925 00:57:48.911216 7124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.911203 7200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.911437 7200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6866565b04f9d85454275bf1dc007, State: Initialized, Role: FOLLOWER
I20260925 00:57:48.911761 7200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.912374 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.912703 7124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:48.912758 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c831f55f840415d854e8af0f14893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "63e32b9ea7414ee38159e2745c5dd09a"
I20260925 00:57:48.913000 7097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:48.910202 6980 master_runner.cc:186] Error getting master registration for 127.16.202.61:38379: OK, instance_id { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" instance_seqno: 1790297868631932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:57:48.914057 6980 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:57:48.914309 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.914436 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.914444 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c831f55f840415d854e8af0f14893bb in term 1.
W20260925 00:57:48.914817 7221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:48.914870 7221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:48.914999 6997 leader_election.cc:302] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c831f55f840415d854e8af0f14893bb, 63e32b9ea7414ee38159e2745c5dd09a; no voters:
I20260925 00:57:48.915192 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.913019 7200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:48.915287 7208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:48.915508 7208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [term 1 LEADER]: Becoming Leader. State: Replica: 5c831f55f840415d854e8af0f14893bb, State: Running, Role: LEADER
I20260925 00:57:48.915856 7208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } }
I20260925 00:57:48.914734 7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c831f55f840415d854e8af0f14893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fd6866565b04f9d85454275bf1dc007"
I20260925 00:57:48.918032 7174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:48.919417 7174 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c831f55f840415d854e8af0f14893bb in term 1.
W20260925 00:57:48.920830 7254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:48.920914 7254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:48.925323 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.915317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.16.202.62:32937)
0925 00:57:48.915448 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.915452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.915959 (+ 507us) server_negotiation.cc:407] Connection header received
0925 00:57:48.916236 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.916240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.916292 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.916344 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.917418 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.918143 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.919374 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.919685 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.919763 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.921841 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.921845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.921848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.921887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.923491 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.923960 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.923963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.923965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.924048 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.924332 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.924335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.924337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.924563 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.924663 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.925060 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.925174 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:57:48.929018 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.929150 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.929330 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.916024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.16.202.61:38379)
0925 00:57:48.916122 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.916126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.920917 (+ 4791us) server_negotiation.cc:407] Connection header received
0925 00:57:48.921119 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.921123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.921177 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.921227 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.921999 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.922737 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.923818 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.924122 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.924189 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.925893 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.925897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.925899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.925925 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.927494 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.927975 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.927979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.927980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.928064 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.928366 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.928370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.928371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.928585 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.928701 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.929077 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.929190 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:57:48.924351 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c831f55f840415d854e8af0f14893bb. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.931102 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:48.931558 7257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:48.932564 7257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:48.935454 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:48.916736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.16.202.60:43135)
0925 00:57:48.916845 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:48.916849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:48.924828 (+ 7979us) server_negotiation.cc:407] Connection header received
0925 00:57:48.924991 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:48.924994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:48.925070 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:48.925118 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:48.926982 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.927751 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.929019 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:48.929351 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:48.929447 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:48.931823 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:48.931827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:48.931831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:48.931860 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:48.933570 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.934066 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.934069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.934071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.934160 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:48.934467 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:48.934470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:48.934472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:48.934695 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:48.934794 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:48.935174 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 00:57:48.935303 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 00:57:48.936146 7239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:48.936232 7239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:48.941242 7174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [term 1 FOLLOWER]: Refusing update from remote peer 5c831f55f840415d854e8af0f14893bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:48.942112 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [term 1 FOLLOWER]: Refusing update from remote peer 5c831f55f840415d854e8af0f14893bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:48.943475 7240 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:48.943970 7240 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:48.948557 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c831f55f840415d854e8af0f14893bb. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.948748 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.956153 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c831f55f840415d854e8af0f14893bb. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.956331 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.963943 7259 mvcc.cc:205] Tried to move back new op lower bound from 7333060071172112384 to 7333060071083520000. Current Snapshot: MvccSnapshot[applied={T|T < 7333060071172112384}]
I20260925 00:57:48.964504 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.964650 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:48.966166 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.966315 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.967280 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.967422 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:48.971233 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c831f55f840415d854e8af0f14893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c831f55f840415d854e8af0f14893bb" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 32937 } } peers { permanent_uuid: "63e32b9ea7414ee38159e2745c5dd09a" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 38379 } } peers { permanent_uuid: "4fd6866565b04f9d85454275bf1dc007" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 43135 } } }
I20260925 00:57:48.971381 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:48.971728 7263 mvcc.cc:205] Tried to move back new op lower bound from 7333060071172112384 to 7333060071083520000. Current Snapshot: MvccSnapshot[applied={T|T < 7333060071172112384}]
I20260925 00:57:48.971963 7261 mvcc.cc:205] Tried to move back new op lower bound from 7333060071172112384 to 7333060071083520000. Current Snapshot: MvccSnapshot[applied={T|T < 7333060071172112384}]
I20260925 00:57:48.973519 7257 catalog_manager.cc:1347] Generated new cluster ID: 00d8f385f07f4f2982e7930da4013c6c
I20260925 00:57:48.973584 7257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:49.005389 7257 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:49.007324 7257 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:49.022799 7257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c831f55f840415d854e8af0f14893bb: Generated new TSK 0
I20260925 00:57:49.023872 7257 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:57:49.060705 7052 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.057165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.16.202.62:32937)
0925 00:57:49.057295 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.057300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.059492 (+ 2192us) server_negotiation.cc:407] Connection header received
0925 00:57:49.059723 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.059728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.059783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.059836 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.060522 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58722: BlockingRecv error: recv got EOF from 127.0.0.1:58722 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 00:57:49.062162 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.057931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.16.202.61:38379)
0925 00:57:49.058057 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.058061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.061242 (+ 3181us) server_negotiation.cc:407] Connection header received
0925 00:57:49.061428 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.061431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.061484 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.061534 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.062044 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45388: BlockingRecv error: recv got EOF from 127.0.0.1:45388 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 00:57:49.063526 7205 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.058655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.16.202.60:43135)
0925 00:57:49.058766 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.058770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.062633 (+ 3863us) server_negotiation.cc:407] Connection header received
0925 00:57:49.062798 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.062801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.062850 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.062897 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.063410 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42450: BlockingRecv error: recv got EOF from 127.0.0.1:42450 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 00:57:49.068499 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.066788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.16.202.61:38379)
0925 00:57:49.066915 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.066919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.067440 (+ 521us) server_negotiation.cc:407] Connection header received
0925 00:57:49.067675 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.067678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.067754 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.067806 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.068393 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45398: BlockingRecv error: recv got EOF from 127.0.0.1:45398 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 00:57:49.073982 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.072255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.16.202.61:38379)
0925 00:57:49.072365 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.072369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.073029 (+ 660us) server_negotiation.cc:407] Connection header received
0925 00:57:49.073218 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.073222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.073274 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.073322 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.073863 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45406: BlockingRecv error: recv got EOF from 127.0.0.1:45406 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 00:57:49.080068 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.078445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.16.202.61:38379)
0925 00:57:49.078562 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.078566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.079093 (+ 527us) server_negotiation.cc:407] Connection header received
0925 00:57:49.079292 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.079296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.079351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.079403 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.079952 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45418: BlockingRecv error: recv got EOF from 127.0.0.1:45418 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 00:57:49.088557 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.086609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.16.202.61:38379)
0925 00:57:49.086808 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.086814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.087542 (+ 728us) server_negotiation.cc:407] Connection header received
0925 00:57:49.087753 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.087756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.087812 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.087873 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.088435 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45428: BlockingRecv error: recv got EOF from 127.0.0.1:45428 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260925 00:57:49.102005 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.100145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.16.202.61:38379)
0925 00:57:49.100272 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.100277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.100927 (+ 650us) server_negotiation.cc:407] Connection header received
0925 00:57:49.101145 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.101149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.101206 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.101262 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.101883 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45430: BlockingRecv error: recv got EOF from 127.0.0.1:45430 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 00:57:49.122874 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.120918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.16.202.61:38379)
0925 00:57:49.121064 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.121069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.121754 (+ 685us) server_negotiation.cc:407] Connection header received
0925 00:57:49.121995 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.121999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.122056 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.122109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.122747 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45440: BlockingRecv error: recv got EOF from 127.0.0.1:45440 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 00:57:49.158591 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.157531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.16.202.61:38379)
0925 00:57:49.157667 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.157671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.157772 (+ 101us) server_negotiation.cc:407] Connection header received
0925 00:57:49.157894 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.157897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.157950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.158034 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.158495 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45454: BlockingRecv error: recv got EOF from 127.0.0.1:45454 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 00:57:49.225878 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.224829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.16.202.61:38379)
0925 00:57:49.224967 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.224970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.225090 (+ 120us) server_negotiation.cc:407] Connection header received
0925 00:57:49.225182 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.225185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.225245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.225328 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.225765 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45470: BlockingRecv error: recv got EOF from 127.0.0.1:45470 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 00:57:49.357055 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.356053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.16.202.61:38379)
0925 00:57:49.356199 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.356203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.356288 (+ 85us) server_negotiation.cc:407] Connection header received
0925 00:57:49.356429 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.356432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.356483 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.356567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.356944 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45474: BlockingRecv error: recv got EOF from 127.0.0.1:45474 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 00:57:49.616508 7130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:49.615398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.16.202.61:38379)
0925 00:57:49.615526 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:49.615529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:49.615637 (+ 108us) server_negotiation.cc:407] Connection header received
0925 00:57:49.615771 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:49.615774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:49.615824 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:49.615906 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:49.616391 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45490: BlockingRecv error: recv got EOF from 127.0.0.1:45490 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:57:49.922235 7254 catalog_manager.cc:1259] Loaded cluster ID: 00d8f385f07f4f2982e7930da4013c6c
I20260925 00:57:49.922323 7254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: loading cluster ID for follower catalog manager: success
I20260925 00:57:49.923985 7254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: acquiring CA information for follower catalog manager: success
I20260925 00:57:49.925211 7254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fd6866565b04f9d85454275bf1dc007: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:57:49.937546 7239 catalog_manager.cc:1259] Loaded cluster ID: 00d8f385f07f4f2982e7930da4013c6c
I20260925 00:57:49.937636 7239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: loading cluster ID for follower catalog manager: success
I20260925 00:57:49.939337 7239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: acquiring CA information for follower catalog manager: success
I20260925 00:57:49.940606 7239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63e32b9ea7414ee38159e2745c5dd09a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:57:50.134927 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:50.130764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.16.202.61:38379)
0925 00:57:50.131029 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:50.131032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:50.131103 (+ 71us) server_negotiation.cc:407] Connection header received
0925 00:57:50.131295 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:50.131299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:50.131446 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:50.131553 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:50.131848 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.132397 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.133361 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.133618 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.133711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:50.133987 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:50.134546 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:50.134641 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:50.134723 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 00:57:50.134777 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260925 00:57:50.142176 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:50.138130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.16.202.60:43135)
0925 00:57:50.138354 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:50.138358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:50.138388 (+ 30us) server_negotiation.cc:407] Connection header received
0925 00:57:50.138557 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:50.138559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:50.138677 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:50.138770 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:50.139011 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.139577 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.140531 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.140843 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.140930 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:50.141304 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:50.141767 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:50.141882 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:50.141948 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 00:57:50.142014 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260925 00:57:50.149235 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:50.145604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.16.202.60:43135)
0925 00:57:50.145724 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:50.145727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:50.145743 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:57:50.145790 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:50.145793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:50.145842 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:50.145907 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:50.146166 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.146902 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.148018 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.148286 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.148394 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:50.148702 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:50.148796 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:50.148843 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:50.149019 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 00:57:50.149088 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:57:50.149287 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:50.146008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.16.202.61:38379)
0925 00:57:50.146135 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:50.146139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:50.146152 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:57:50.146205 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:50.146209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:50.146256 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:50.146318 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:50.146578 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.147129 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.148177 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.148493 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.148584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:50.148866 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:50.148948 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:50.149016 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:50.149096 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 00:57:50.149156 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:57:50.150296 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:50.145394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.16.202.62:32937)
0925 00:57:50.145802 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:50.145807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:50.145825 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:57:50.146197 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:50.146201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:50.146388 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:50.146514 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:50.146735 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.147495 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.148487 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:50.148821 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:50.148918 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:50.149532 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:50.149925 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:50.150006 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:50.150094 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 00:57:50.150151 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":72,"threads_started":1}
I20260925 00:57:50.158409 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 6980
I20260925 00:57:50.163967 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 7055
I20260925 00:57:50.169261 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 7132
2026-09-25T00:57:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3027 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:57:50 dist-test-slave-rpk1 krb5kdc[7350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:57:50 dist-test-slave-rpk1 krb5kdc[7350](info): set up 2 sockets
Sep 25 00:57:50 dist-test-slave-rpk1 krb5kdc[7350](info): commencing operation
krb5kdc: starting...
W20260925 00:57:51.129065 17192 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/test-user.keytab.
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.202.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
Entry for principal kudu/127.16.202.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab.
I20260925 00:57:51.257236 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39599
--webserver_interface=127.16.202.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39599,127.16.202.61:33695,127.16.202.60:39083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:57:51.369710 7363 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:57:51.370071 7363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:51.370119 7363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:51.370184 7363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:51.374715 7363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:51.374815 7363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:51.374850 7363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:51.374873 7363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:51.374892 7363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:51.380564 7363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39599,127.16.202.61:33695,127.16.202.60:39083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.62.keytab
--principal=kudu/127.16.202.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.62:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.202.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:51.381781 7363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:51.382825 7363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:51.391783 7369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:51.391798 7368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:51.391855 7371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:51.391970 7363 server_base.cc:1034] running on GCE node
I20260925 00:57:51.392664 7363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:51.393829 7363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:51.395012 7363 hybrid_clock.cc:648] HybridClock initialized: now 1790297871394995 us; error 33 us; skew 500 ppm
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:51.397827 7363 init.cc:377] Logged in from keytab as kudu/127.16.202.62@KRBTEST.COM (short username kudu)
I20260925 00:57:51.398729 7363 webserver.cc:460] Webserver started at http://127.16.202.62:40897/ using document root <none> and password file <none>
I20260925 00:57:51.399281 7363 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:51.399410 7363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:51.399641 7363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:51.402029 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data/instance:
uuid: "0418868597ef4e8bae76d224abd55543"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "af8202d7c22aa74b99d82d978364218c"
server_key_iv: "b7c5372648b6b55072df93a398f05992"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.402640 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal/instance:
uuid: "0418868597ef4e8bae76d224abd55543"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "af8202d7c22aa74b99d82d978364218c"
server_key_iv: "b7c5372648b6b55072df93a398f05992"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.406667 7363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:57:51.409260 7378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:51.410600 7363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:51.410729 7363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal
uuid: "0418868597ef4e8bae76d224abd55543"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "af8202d7c22aa74b99d82d978364218c"
server_key_iv: "b7c5372648b6b55072df93a398f05992"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.410838 7363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:51.448405 7363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:51.451942 7363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:51.452160 7363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:51.459103 7363 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.62:39599
I20260925 00:57:51.459390 7430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.62:39599 every 8 connection(s)
I20260925 00:57:51.460079 7363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/data/info.pb
I20260925 00:57:51.462893 7431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:51.464498 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 7363
I20260925 00:57:51.464617 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-0/wal/instance
I20260925 00:57:51.464946 17192 external_mini_cluster.cc:1428] Setting key 85a828fde8008d61b3f207bda94e0ba6
I20260925 00:57:51.467386 7431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.202.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.61@KRBTEST.COM" created.
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:51.482378 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.469773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.16.202.62:39599)
0925 00:57:51.470199 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.470217 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.470470 (+ 253us) server_negotiation.cc:407] Connection header received
0925 00:57:51.471468 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.471472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.471663 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.471819 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.472648 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.473617 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.474565 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.474913 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.475051 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.477522 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.477530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.477544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.477571 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.479253 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.479731 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.479737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.479739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.479848 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.480139 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.480142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.480144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.480572 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.480718 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.481064 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.481304 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260925 00:57:51.482378 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.469774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39599 (local address 127.0.0.1:50084)
0925 00:57:51.470324 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.470361 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.471173 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.471832 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.471837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.472262 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.472513 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.472530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.473726 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.473729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.474458 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.474468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.475055 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.475059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.475191 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.475705 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.475722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.477346 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.479372 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.479382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.479394 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.479624 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.479933 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.479935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.479937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.480054 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.480701 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.480708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.480837 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.481000 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.481369 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":102,"spinlock_wait_cycles":19328,"thread_start_us":169,"threads_started":1}
I20260925 00:57:51.484921 7431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:51.485729 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.485412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52878)
0925 00:57:51.485575 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.485654 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 00:57:51.485915 7379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
I20260925 00:57:51.486842 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.486627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52882)
0925 00:57:51.486731 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.486778 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 00:57:51.486999 7431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:33695: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab.
Entry for principal kudu/127.16.202.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab.
I20260925 00:57:51.492322 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:33695
--webserver_interface=127.16.202.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab
--principal=kudu/127.16.202.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39599,127.16.202.61:33695,127.16.202.60:39083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:57:51.536289 7431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } attempt: 1
I20260925 00:57:51.537271 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.536928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52884)
0925 00:57:51.537103 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.537177 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 00:57:51.538110 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.537840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52888)
0925 00:57:51.537952 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.538008 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:57:51.538282 7431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:33695: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
I20260925 00:57:51.606480 7431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } attempt: 2
I20260925 00:57:51.607429 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.607077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52894)
0925 00:57:51.607249 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.607327 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 00:57:51.608284 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.608009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52902)
0925 00:57:51.608135 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.608197 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 00:57:51.608484 7431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.61:33695: Network error: Client connection negotiation failed: client connection to 127.16.202.61:33695: connect: Connection refused (error 111)
W20260925 00:57:51.613437 7438 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:57:51.613773 7438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:51.613801 7438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:51.613849 7438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:51.618106 7438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:51.618176 7438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:51.618196 7438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:51.618212 7438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:51.618227 7438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:51.623804 7438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39599,127.16.202.61:33695,127.16.202.60:39083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.61.keytab
--principal=kudu/127.16.202.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.61:33695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.202.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:51.624945 7438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:51.625969 7438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:51.634868 7444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:51.634868 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:51.634893 7443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:51.635121 7438 server_base.cc:1034] running on GCE node
I20260925 00:57:51.635664 7438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:51.636811 7438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:51.637996 7438 hybrid_clock.cc:648] HybridClock initialized: now 1790297871637976 us; error 35 us; skew 500 ppm
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:51.640884 7438 init.cc:377] Logged in from keytab as kudu/127.16.202.61@KRBTEST.COM (short username kudu)
I20260925 00:57:51.641813 7438 webserver.cc:460] Webserver started at http://127.16.202.61:33065/ using document root <none> and password file <none>
I20260925 00:57:51.642347 7438 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:51.642407 7438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:51.642601 7438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:51.644696 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data/instance:
uuid: "b7b3a6047158408a8eed058d572ed841"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "8bc32ec5640bbad77b64742d9374fe71"
server_key_iv: "7e4c732f625993ad53ba77bf669fe93d"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.645318 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal/instance:
uuid: "b7b3a6047158408a8eed058d572ed841"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "8bc32ec5640bbad77b64742d9374fe71"
server_key_iv: "7e4c732f625993ad53ba77bf669fe93d"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.649273 7438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:57:51.652273 7453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:51.653790 7438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:51.653925 7438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal
uuid: "b7b3a6047158408a8eed058d572ed841"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "8bc32ec5640bbad77b64742d9374fe71"
server_key_iv: "7e4c732f625993ad53ba77bf669fe93d"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.654055 7438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:51.684932 7438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:51.688316 7438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:51.688515 7438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:51.695549 7438 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.61:33695
I20260925 00:57:51.695819 7505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.61:33695 every 8 connection(s)
I20260925 00:57:51.696607 7438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/data/info.pb
I20260925 00:57:51.699666 7506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:51.699674 7431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } attempt: 3
I20260925 00:57:51.701292 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 7438
I20260925 00:57:51.701440 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-1/wal/instance
I20260925 00:57:51.701803 17192 external_mini_cluster.cc:1428] Setting key a1e904ef4e2190fd514e5e07b95ed45b
I20260925 00:57:51.705178 7506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.202.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.202.60@KRBTEST.COM" created.
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:51.717717 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.707235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.16.202.62:39599)
0925 00:57:51.707372 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.707377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.707760 (+ 383us) server_negotiation.cc:407] Connection header received
0925 00:57:51.707964 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.707967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.708017 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.708060 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.708976 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.709735 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.710628 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.711316 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.711414 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.714106 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.714110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.714113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.714140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.715591 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.716089 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.716093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.716094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.716179 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.716483 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.716486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.716488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.716729 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.716823 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.717409 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.717549 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:57:51.717839 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.700208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52918)
0925 00:57:51.700390 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.700410 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.700529 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.703626 (+ 3097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.703630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.703868 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.704030 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.704040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.706415 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.706420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.707273 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.707288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.708415 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.708419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.708548 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.709136 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.709151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.713454 (+ 4303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.715833 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.715836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.715838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.716096 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.716466 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.716469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.716471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.716557 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.717304 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.717307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.717414 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.717496 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.717641 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 00:57:51.719038 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.701352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.16.202.61:33695)
0925 00:57:51.701914 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.701934 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.701981 (+ 47us) server_negotiation.cc:407] Connection header received
0925 00:57:51.702676 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.702702 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.703214 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.704270 (+ 1056us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.704738 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.706231 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.707466 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.708263 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.708454 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.713658 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.713666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.713683 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.713714 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.715699 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.716227 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.716233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.716234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.716365 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.716650 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.716653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.716654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.717181 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.718057 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.718167 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.718320 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":373,"spinlock_wait_cycles":1280,"thread_start_us":162,"threads_started":1}
I20260925 00:57:51.719034 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.707172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39599 (local address 127.0.0.1:50098)
0925 00:57:51.707632 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.707686 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.707848 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.708210 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.708215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.708669 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.708859 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.708866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.709855 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.709859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.710505 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.710516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.711484 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.711489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.711603 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.712212 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.712235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.713962 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.715734 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.715743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.715753 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.715989 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.716263 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.716267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.716269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.716393 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.716834 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.716842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.717171 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.717330 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.717690 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":41,"spinlock_wait_cycles":1920,"thread_start_us":132,"threads_started":1}
I20260925 00:57:51.720486 7506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:51.721217 7431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:51.721906 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.721673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50844)
0925 00:57:51.721788 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.721838 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:57:51.722754 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.722465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50852)
0925 00:57:51.722635 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.722689 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 00:57:51.722882 7431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:39083: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:57:51.729532 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.721172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.16.202.61:33695)
0925 00:57:51.721273 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.721277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.721362 (+ 85us) server_negotiation.cc:407] Connection header received
0925 00:57:51.721803 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.721806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.721868 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.721951 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.722364 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.723063 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.723888 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.724111 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.724186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.726392 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.726395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.726397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.726419 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.727882 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.728323 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.728325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.728326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.728395 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.728666 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.728669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.728702 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.728895 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.728980 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.729189 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.729298 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 00:57:51.729539 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.721045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52924)
0925 00:57:51.721300 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.721315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.721405 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.721990 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.721993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.722164 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.722267 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.722275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.723202 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.723205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.723758 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.723766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.724219 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.724222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.724290 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.724921 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.724934 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.726236 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.727994 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.727997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.727999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.728239 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.728502 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.728504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.728506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.728589 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.728981 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.728984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.729069 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.729153 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.729365 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":52}
I20260925 00:57:51.730374 7506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:51.731140 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.730830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50868)
0925 00:57:51.730980 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.731068 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.202.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab.
Entry for principal kudu/127.16.202.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab.
W20260925 00:57:51.731808 7455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
I20260925 00:57:51.732599 17192 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
/tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:39083
--webserver_interface=127.16.202.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab
--principal=kudu/127.16.202.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.202.20:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.202.62:39599,127.16.202.61:33695,127.16.202.60:39083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:57:51.732846 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.732580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50884)
0925 00:57:51.732718 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.732772 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:57:51.733001 7506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:39083: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
I20260925 00:57:51.754079 7431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } attempt: 1
I20260925 00:57:51.755038 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.754631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50888)
0925 00:57:51.754841 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.754923 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 00:57:51.756039 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.755702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50904)
0925 00:57:51.755816 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.755954 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:57:51.756222 7431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:39083: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
I20260925 00:57:51.782428 7506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } attempt: 1
I20260925 00:57:51.783445 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.783059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50906)
0925 00:57:51.783240 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.783306 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 00:57:51.784264 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.784013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50920)
0925 00:57:51.784143 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.784190 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 00:57:51.784427 7506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:39083: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
I20260925 00:57:51.831415 7431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } attempt: 2
I20260925 00:57:51.832332 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.832008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50928)
0925 00:57:51.832182 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.832247 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 00:57:51.833310 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.833065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50930)
0925 00:57:51.833193 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.833243 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 00:57:51.833453 7431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:39083: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
I20260925 00:57:51.852638 7506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } attempt: 2
I20260925 00:57:51.853531 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.853238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50946)
0925 00:57:51.853384 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.853451 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 00:57:51.854229 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.854028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50956)
0925 00:57:51.854125 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.854171 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 00:57:51.854374 7506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.202.60:39083: Network error: Client connection negotiation failed: client connection to 127.16.202.60:39083: connect: Connection refused (error 111)
W20260925 00:57:51.856390 7513 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:57:51.856717 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:57:51.856760 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:57:51.856802 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:57:51.861120 7513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:57:51.861191 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:57:51.861210 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:57:51.861227 7513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:57:51.861241 7513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:57:51.867192 7513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.202.20:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.202.62:39599,127.16.202.61:33695,127.16.202.60:39083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/krb5kdc/kudu_127.16.202.60.keytab
--principal=kudu/127.16.202.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.202.60:39083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.202.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:57:51.868304 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:57:51.869371 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:57:51.878029 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:57:51.878032 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:51.878180 7513 server_base.cc:1034] running on GCE node
W20260925 00:57:51.878365 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:57:51.878974 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:57:51.880100 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:57:51.881276 7513 hybrid_clock.cc:648] HybridClock initialized: now 1790297871881251 us; error 42 us; skew 500 ppm
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:57:51.884153 7513 init.cc:377] Logged in from keytab as kudu/127.16.202.60@KRBTEST.COM (short username kudu)
I20260925 00:57:51.885229 7513 webserver.cc:460] Webserver started at http://127.16.202.60:34537/ using document root <none> and password file <none>
I20260925 00:57:51.885802 7513 fs_manager.cc:359] Metadata directory not provided
I20260925 00:57:51.885869 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:57:51.886073 7513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:57:51.888198 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data/instance:
uuid: "5953b88f0b2f4c3a8e0ba71af35bf276"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "4806b5d51929ae6b2ca4ca287548660d"
server_key_iv: "5932382490e1a5cf98dbc027b22fd3b5"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.888880 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal/instance:
uuid: "5953b88f0b2f4c3a8e0ba71af35bf276"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "4806b5d51929ae6b2ca4ca287548660d"
server_key_iv: "5932382490e1a5cf98dbc027b22fd3b5"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.892963 7513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:57:51.895555 7528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:57:51.896903 7513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:57:51.897053 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal
uuid: "5953b88f0b2f4c3a8e0ba71af35bf276"
format_stamp: "Formatted at 2026-09-25 00:57:51 on dist-test-slave-rpk1"
server_key: "4806b5d51929ae6b2ca4ca287548660d"
server_key_iv: "5932382490e1a5cf98dbc027b22fd3b5"
server_key_version: "encryptionkey@0"
I20260925 00:57:51.897161 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:57:51.911438 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:57:51.914968 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:57:51.915164 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:57:51.922492 7513 rpc_server.cc:307] RPC server started. Bound to: 127.16.202.60:39083
I20260925 00:57:51.922758 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.202.60:39083 every 8 connection(s)
I20260925 00:57:51.923553 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/data/info.pb
I20260925 00:57:51.926587 7581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:57:51.929961 17192 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu as pid 7513
I20260925 00:57:51.930096 17192 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPY1ZZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297591180183-17192-0/minicluster-data/master-2/wal/instance
I20260925 00:57:51.930397 17192 external_mini_cluster.cc:1428] Setting key 622c9fff33038441068ee0025f624c27
I20260925 00:57:51.930745 7581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:51.932641 7431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } attempt: 3
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.62@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
I20260925 00:57:51.943811 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.933051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50986)
0925 00:57:51.933253 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.933267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.933352 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.933898 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.933901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.934107 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.934207 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.934214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.935776 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.935779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.936404 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.936412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.937375 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.937379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.937453 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.937956 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.937972 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.939478 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.941731 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.941735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.941738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.942040 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.942407 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.942413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.942504 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.943344 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.943347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.943452 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.943515 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.943637 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 00:57:51.943964 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.932754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.16.202.62:39599)
0925 00:57:51.932928 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.932932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.933368 (+ 436us) server_negotiation.cc:407] Connection header received
0925 00:57:51.933758 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.933761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.933820 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.933894 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.934875 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.935615 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.937014 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.937221 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.937308 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.940352 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.940356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.940358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.940387 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.942040 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942579 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.942583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.942584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.942677 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942938 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.942949 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.942951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.943180 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.943269 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.943735 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.943847 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 00:57:51.945084 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.933428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.16.202.60:39083)
0925 00:57:51.933684 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.933692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.933704 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:57:51.933735 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.933737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.933773 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.933902 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.934329 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.935627 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.936514 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.937204 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.937366 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.939620 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.939626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.939637 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.939660 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.941619 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942146 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.942152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.942159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.942296 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942604 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.942607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.942609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.943212 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.944183 (+ 971us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.944288 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.944428 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260925 00:57:51.945089 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.932663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.62:39599 (local address 127.0.0.1:50110)
0925 00:57:51.933243 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.933297 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.933551 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.934072 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.934078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.934522 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.934717 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.934732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.935755 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.935758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.936909 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.936919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.937690 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.937693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.937770 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.938234 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.938253 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.940189 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.942157 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.942174 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.942465 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.942769 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.942772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.942774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.942856 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.943283 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.943291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.943464 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.943665 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.944019 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260925 00:57:51.945549 7506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } attempt: 3
I20260925 00:57:51.945966 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.931722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.16.202.60:39083)
0925 00:57:51.932157 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.932171 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.932205 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:57:51.933156 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.933180 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.933586 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.934691 (+ 1105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.934765 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.935616 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.936915 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.937258 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.937713 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.941099 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.941103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.941105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.941130 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.943236 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.943723 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.943726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.943728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.943803 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.944490 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.944493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.944495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.944725 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.945644 (+ 919us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.945720 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.945829 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":271,"server-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260925 00:57:51.947499 7581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:57:51.950750 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543: Bootstrap starting.
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.61@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.62@KRBTEST.COM
I20260925 00:57:51.953935 7431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543: Neither blocks nor log segments found. Creating new log.
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
I20260925 00:57:51.954788 7431 log.cc:824] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:51.955224 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.946022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:50996)
0925 00:57:51.946199 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.946213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.946301 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.946559 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.946562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.946741 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.946867 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.946875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.947887 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.947891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.948653 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.948664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.949312 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.949316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.949413 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.949903 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.949917 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.951493 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.953402 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.953406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.953408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.953672 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.954092 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.954096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.954098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.954185 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.954670 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.954673 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.954777 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.954898 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.955054 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 00:57:51.955252 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.946075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.16.202.60:39083)
0925 00:57:51.946158 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.946162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.946252 (+ 90us) server_negotiation.cc:407] Connection header received
0925 00:57:51.946376 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.946378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.946411 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.946525 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.946999 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.947765 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.948827 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.949174 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.949642 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.951635 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.951639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.951641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.951667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.953257 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.953893 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.953897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.953899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.953982 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.954303 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.954306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.954308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.954529 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.954659 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.954970 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.955093 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 00:57:51.957202 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543: No bootstrap required, opened a new log
I20260925 00:57:51.959149 7506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: Bootstrap starting.
I20260925 00:57:51.960055 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.948133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.16.202.62:39599)
0925 00:57:51.948490 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.948494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.948506 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:57:51.948537 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.948540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.948583 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.948659 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.949526 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.950221 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.951334 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.951642 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.951728 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.955074 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.955078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.955081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.955107 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.957983 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.958545 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.958549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.958551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.958646 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.958970 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.958974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.958976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.959357 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.959555 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.959774 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.959898 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260925 00:57:51.961186 7431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:51.961432 7431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0418868597ef4e8bae76d224abd55543, State: Initialized, Role: FOLLOWER
I20260925 00:57:51.961783 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:51.961966 7506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:51.962913 7506 log.cc:824] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:51.963308 7431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:51.963999 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.948177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.16.202.61:33695)
0925 00:57:51.948282 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.948286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.948431 (+ 145us) server_negotiation.cc:407] Connection header received
0925 00:57:51.948789 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.948792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.948847 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.948900 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.949492 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.950549 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.951542 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.951839 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.952364 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.955741 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.955745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.955748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.955771 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.957331 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.962472 (+ 5141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.962476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.962478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.962579 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.962947 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.962951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.962952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.963245 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.963346 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.963715 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.963842 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":43}
I20260925 00:57:51.964668 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.947955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.61:33695 (local address 127.0.0.1:52928)
0925 00:57:51.948366 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.948379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.948630 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.949041 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.949044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.949231 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.949347 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.949355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.950687 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.950690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.951410 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.951422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.952754 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.952758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.952860 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.953265 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.953278 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.955591 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.960765 (+ 5174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.960769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.960771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.962324 (+ 1553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.962703 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.962706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.962708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.962828 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.963350 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.963353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.963519 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.964336 (+ 817us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.964516 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346}
I20260925 00:57:51.965235 7506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: No bootstrap required, opened a new log
I20260925 00:57:51.967634 7589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:51.967761 7589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.202.61@KRBTEST.COM
W20260925 00:57:51.969349 7601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 00:57:51.969408 7581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } has no permanent_uuid. Determining permanent_uuid...
W20260925 00:57:51.969481 7601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:51.973224 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.961733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.16.202.61:33695)
0925 00:57:51.962069 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.962079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.962093 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:57:51.962130 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.962133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.962182 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.963033 (+ 851us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.963130 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.964142 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.965210 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.965552 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.965996 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.968528 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.968531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.968534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.968559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.970870 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.971369 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.971372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.971374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.971472 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.971761 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.971764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.971766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.971972 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.972821 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.972900 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.973028 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260925 00:57:51.969646 7363 master_runner.cc:186] Error getting master registration for 127.16.202.61:33695: OK, instance_id { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" instance_seqno: 1790297871693536 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:57:51.973838 7363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:57:51.974805 7506 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:51.975085 7506 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b3a6047158408a8eed058d572ed841, State: Initialized, Role: FOLLOWER
I20260925 00:57:51.975473 7506 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
Sep 25 00:57:51 dist-test-slave-rpk1 krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297871, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.202.60@KRBTEST.COM for kudu/127.16.202.60@KRBTEST.COM
I20260925 00:57:51.976101 7603 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:51.976305 7603 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:51.977156 7506 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:51.977804 17192 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:57:51.979511 7405 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0418868597ef4e8bae76d224abd55543"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50114
I20260925 00:57:51.979794 7405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 00:57:51.979905 7405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:51.980373 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.970130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.202.60:39083 (local address 127.0.0.1:51008)
0925 00:57:51.970452 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 00:57:51.970467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:57:51.970546 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:57:51.970837 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:57:51.970840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:57:51.971015 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:57:51.971110 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.971118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.972116 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.972119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.972960 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:57:51.972972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.973899 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.973904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:57:51.973988 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.974588 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:57:51.974603 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:57:51.976946 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:57:51.978818 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.978821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.978823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.979033 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.979308 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:57:51.979310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:57:51.979312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:57:51.979406 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:57:51.979858 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:57:51.979861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:57:51.979946 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 00:57:51.980045 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:57:51.980204 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260925 00:57:51.980885 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.970329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.16.202.60:39083)
0925 00:57:51.970543 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.970548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.970564 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:57:51.970638 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.970641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.970677 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.970733 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.971226 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.972002 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.973080 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.973412 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.973506 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.977096 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.977099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.977102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.977127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.978708 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.979126 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.979130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.979132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.979214 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.979498 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.979501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.979502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.979715 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.979820 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.980584 (+ 764us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.980748 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:57:51.981424 7405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:51.983464 7405 leader_election.cc:288] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [CANDIDATE]: Term 1 election: Requested vote from peers b7b3a6047158408a8eed058d572ed841 (127.16.202.61:33695), 5953b88f0b2f4c3a8e0ba71af35bf276 (127.16.202.60:39083)
I20260925 00:57:51.983140 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0418868597ef4e8bae76d224abd55543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7b3a6047158408a8eed058d572ed841"
I20260925 00:57:51.984879 7480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:57:51.986306 7480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0418868597ef4e8bae76d224abd55543 in term 1.
I20260925 00:57:51.986724 7379 leader_election.cc:302] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0418868597ef4e8bae76d224abd55543, b7b3a6047158408a8eed058d572ed841; no voters:
I20260925 00:57:51.987120 7589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:57:51.987365 7589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [term 1 LEADER]: Becoming Leader. State: Replica: 0418868597ef4e8bae76d224abd55543, State: Running, Role: LEADER
I20260925 00:57:51.987744 7589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:51.990505 7581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: Bootstrap starting.
I20260925 00:57:51.995966 7581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: Neither blocks nor log segments found. Creating new log.
I20260925 00:57:51.997035 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.985746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.16.202.62:39599)
0925 00:57:51.985846 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.985851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.986899 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 00:57:51.987068 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.987071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.987120 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.987169 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.988333 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.989036 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.990193 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.990503 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.990582 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.992423 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.992427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.992430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.992457 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.994206 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.995361 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.995365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.995367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.995464 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.995757 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.995760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.995762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.995985 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.996226 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.996754 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.996873 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 00:57:51.999791 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.986070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.16.202.61:33695)
0925 00:57:51.986174 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.986178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.986372 (+ 194us) server_negotiation.cc:407] Connection header received
0925 00:57:51.986607 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.986610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.986659 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.986706 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.987767 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.988573 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.990118 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.990496 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.990575 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.993358 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.993362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.993365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.993403 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.995234 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.997325 (+ 2091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.997329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.997331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.997434 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.998951 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.998954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.998956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.999145 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.999279 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:51.999536 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 00:57:51.999652 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 00:57:51.999822 7581 log.cc:824] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: Log is configured to *not* fsync() on all Append() calls
I20260925 00:57:51.998538 7438 master_runner.cc:186] Error getting master registration for 127.16.202.60:39083: OK, instance_id { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" instance_seqno: 1790297871920371 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:57:52.002099 7438 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:57:52.002734 7581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: No bootstrap required, opened a new log
I20260925 00:57:52.004032 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:51.987760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.16.202.60:39083)
0925 00:57:51.987874 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:51.987878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:51.988811 (+ 933us) server_negotiation.cc:407] Connection header received
0925 00:57:51.988973 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:51.988977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:51.989026 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:51.989075 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:51.990721 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.991430 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.992487 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:51.992800 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:51.992878 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:51.994949 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:57:51.994953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:57:51.994956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:57:51.994986 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:57:51.996493 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.997811 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.997814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.997816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.997904 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:57:51.998180 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:57:51.998184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:57:51.998186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:57:51.998409 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:57:51.998507 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:52.003708 (+ 5201us) server_negotiation.cc:299] Negotiation successful
0925 00:57:52.003870 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:57:52.005008 7621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0418868597ef4e8bae76d224abd55543. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.005189 7621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:52.005779 7625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:57:52.006875 7581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:52.007148 7581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5953b88f0b2f4c3a8e0ba71af35bf276, State: Initialized, Role: FOLLOWER
I20260925 00:57:52.007498 7581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } }
I20260925 00:57:52.007980 7625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:57:52.008952 7581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:57:52.009591 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0418868597ef4e8bae76d224abd55543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276"
I20260925 00:57:52.009846 7555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [term 0 FOLLOWER]: Advancing to term 1
W20260925 00:57:52.010597 7614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:52.010663 7614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:52.011210 7555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0418868597ef4e8bae76d224abd55543 in term 1.
I20260925 00:57:52.013691 7555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [term 1 FOLLOWER]: Refusing update from remote peer 0418868597ef4e8bae76d224abd55543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:52.013861 7626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.013988 7626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:52.014034 7480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [term 1 FOLLOWER]: Refusing update from remote peer 0418868597ef4e8bae76d224abd55543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:57:52.014331 7621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:57:52.014904 7621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:57:52.018695 7639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:57:52.018800 7639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:57:52.019011 7603 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0418868597ef4e8bae76d224abd55543. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.019150 7603 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:52.020939 7626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0418868597ef4e8bae76d224abd55543. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.021078 7626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:52.022521 7621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.022657 7621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:52.025110 7625 catalog_manager.cc:1347] Generated new cluster ID: 553d1f2f03454dacafefa6441ae43e23
I20260925 00:57:52.025199 7625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:57:52.025616 7603 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.025731 7603 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:52.026188 7589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.026312 7589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543 [sys.catalog]: This master's current role is: LEADER
I20260925 00:57:52.029680 7626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0418868597ef4e8bae76d224abd55543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0418868597ef4e8bae76d224abd55543" member_type: VOTER last_known_addr { host: "127.16.202.62" port: 39599 } } peers { permanent_uuid: "b7b3a6047158408a8eed058d572ed841" member_type: VOTER last_known_addr { host: "127.16.202.61" port: 33695 } } peers { permanent_uuid: "5953b88f0b2f4c3a8e0ba71af35bf276" member_type: VOTER last_known_addr { host: "127.16.202.60" port: 39083 } } }
I20260925 00:57:52.029826 7626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:57:52.031546 7641 mvcc.cc:205] Tried to move back new op lower bound from 7333060083761483776 to 7333060083682422784. Current Snapshot: MvccSnapshot[applied={T|T < 7333060083761483776}]
I20260925 00:57:52.080765 7625 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:57:52.081794 7625 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:57:52.112226 7625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0418868597ef4e8bae76d224abd55543: Generated new TSK 0
I20260925 00:57:52.113250 7625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:57:52.152621 7434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.149937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.16.202.62:39599)
0925 00:57:52.150074 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.150078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.151423 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 00:57:52.151638 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.151642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.151698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.151751 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.152473 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50128: BlockingRecv error: recv got EOF from 127.0.0.1:50128 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 00:57:52.154098 7584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.150835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.16.202.60:39083)
0925 00:57:52.151036 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.151054 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.153174 (+ 2120us) server_negotiation.cc:407] Connection header received
0925 00:57:52.153344 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.153348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.153412 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.153464 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.153974 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51024: BlockingRecv error: recv got EOF from 127.0.0.1:51024 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 00:57:52.156023 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.154597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.16.202.61:33695)
0925 00:57:52.154709 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.154713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.155174 (+ 461us) server_negotiation.cc:407] Connection header received
0925 00:57:52.155329 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.155332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.155388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.155437 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.155908 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52956: BlockingRecv error: recv got EOF from 127.0.0.1:52956 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 00:57:52.160405 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.158849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.16.202.61:33695)
0925 00:57:52.158966 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.158970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.159428 (+ 458us) server_negotiation.cc:407] Connection header received
0925 00:57:52.159644 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.159647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.159720 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.159773 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.160286 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52972: BlockingRecv error: recv got EOF from 127.0.0.1:52972 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 00:57:52.165841 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.164358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.16.202.61:33695)
0925 00:57:52.164487 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.164492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.164835 (+ 343us) server_negotiation.cc:407] Connection header received
0925 00:57:52.165061 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.165065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.165121 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.165175 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.165724 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52974: BlockingRecv error: recv got EOF from 127.0.0.1:52974 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 00:57:52.171826 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.170262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.16.202.61:33695)
0925 00:57:52.170372 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.170376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.170881 (+ 505us) server_negotiation.cc:407] Connection header received
0925 00:57:52.171075 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.171079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.171133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.171184 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.171715 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52978: BlockingRecv error: recv got EOF from 127.0.0.1:52978 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 00:57:52.180202 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.178435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.16.202.61:33695)
0925 00:57:52.178591 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.178596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.179157 (+ 561us) server_negotiation.cc:407] Connection header received
0925 00:57:52.179391 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.179396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.179453 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.179510 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.180075 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52984: BlockingRecv error: recv got EOF from 127.0.0.1:52984 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 00:57:52.194312 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.192293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.16.202.61:33695)
0925 00:57:52.192447 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.192451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.193153 (+ 702us) server_negotiation.cc:407] Connection header received
0925 00:57:52.193411 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.193415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.193477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.193538 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.194180 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52990: BlockingRecv error: recv got EOF from 127.0.0.1:52990 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 00:57:52.214798 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.213148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.16.202.61:33695)
0925 00:57:52.213290 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.213295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.213631 (+ 336us) server_negotiation.cc:407] Connection header received
0925 00:57:52.213830 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.213834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.213888 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.213958 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.214672 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53006: BlockingRecv error: recv got EOF from 127.0.0.1:53006 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 00:57:52.250890 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.249729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.16.202.61:33695)
0925 00:57:52.249886 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.249890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.250018 (+ 128us) server_negotiation.cc:407] Connection header received
0925 00:57:52.250109 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.250112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.250164 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.250255 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.250782 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53016: BlockingRecv error: recv got EOF from 127.0.0.1:53016 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 00:57:52.318148 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.316964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.16.202.61:33695)
0925 00:57:52.317168 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.317171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.317285 (+ 114us) server_negotiation.cc:407] Connection header received
0925 00:57:52.317388 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.317390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.317443 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.317535 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.318021 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53032: BlockingRecv error: recv got EOF from 127.0.0.1:53032 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260925 00:57:52.449522 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.448089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.16.202.61:33695)
0925 00:57:52.448275 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.448280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.448407 (+ 127us) server_negotiation.cc:407] Connection header received
0925 00:57:52.448579 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.448583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.448638 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.448728 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.449393 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53042: BlockingRecv error: recv got EOF from 127.0.0.1:53042 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 00:57:52.708618 7588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:57:52.707581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.16.202.61:33695)
0925 00:57:52.707710 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:52.707714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:52.707784 (+ 70us) server_negotiation.cc:407] Connection header received
0925 00:57:52.707904 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:52.707907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:52.707973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:52.708067 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:57:52.708498 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37990: BlockingRecv error: recv got EOF from 127.0.0.1:37990 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:57:53.011895 7614 catalog_manager.cc:1259] Loaded cluster ID: 553d1f2f03454dacafefa6441ae43e23
I20260925 00:57:53.011996 7614 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: loading cluster ID for follower catalog manager: success
I20260925 00:57:53.013744 7614 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: acquiring CA information for follower catalog manager: success
I20260925 00:57:53.015008 7614 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7b3a6047158408a8eed058d572ed841: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:57:53.020143 7639 catalog_manager.cc:1259] Loaded cluster ID: 553d1f2f03454dacafefa6441ae43e23
I20260925 00:57:53.020227 7639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: loading cluster ID for follower catalog manager: success
I20260925 00:57:53.021895 7639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: acquiring CA information for follower catalog manager: success
I20260925 00:57:53.023120 7639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5953b88f0b2f4c3a8e0ba71af35bf276: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:57:53.226748 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:53.222775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.16.202.61:33695)
0925 00:57:53.223059 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:53.223063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:53.223079 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:57:53.223214 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:53.223218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:53.223349 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:53.223450 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:53.223760 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.224296 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.225214 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.225442 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.225534 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:53.225862 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:53.226335 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:53.226445 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:53.226527 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 00:57:53.226589 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":65,"threads_started":1}
I20260925 00:57:53.233753 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:53.229862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.16.202.60:39083)
0925 00:57:53.230113 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:53.230116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:53.230131 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:57:53.230210 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:53.230212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:53.230344 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:53.230447 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:53.230683 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.231234 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.232092 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.232331 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.232423 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:53.232847 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:53.233341 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:53.233444 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:53.233543 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 00:57:53.233598 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260925 00:57:53.241528 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:53.237149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.16.202.60:39083)
0925 00:57:53.237292 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:53.237297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:53.237797 (+ 500us) server_negotiation.cc:407] Connection header received
0925 00:57:53.237853 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:53.237855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:53.237907 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:53.237972 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:53.238436 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.239286 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.240387 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.240630 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.240759 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:53.241070 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:53.241153 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:53.241214 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:53.241326 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 00:57:53.241385 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:57:53.241858 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:53.237025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.16.202.61:33695)
0925 00:57:53.237295 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:53.237300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:53.238068 (+ 768us) server_negotiation.cc:407] Connection header received
0925 00:57:53.238127 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:53.238130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:53.238186 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:53.238248 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:53.238488 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.239285 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.240449 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.240784 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.240878 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:53.241116 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:53.241195 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:53.241257 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:53.241693 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 00:57:53.241752 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260925 00:57:53.242394 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:57:53.237121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.16.202.62:39599)
0925 00:57:53.237361 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 00:57:53.237365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:57:53.237532 (+ 167us) server_negotiation.cc:407] Connection header received
0925 00:57:53.237687 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:57:53.237689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:57:53.237842 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:57:53.237982 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:57:53.238216 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.239022 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.240202 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:57:53.240528 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:57:53.240625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:57:53.241013 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:57:53.241398 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:57:53.241486 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 00:57:53.241840 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 00:57:53.241905 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260925 00:57:53.245438 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 7363
I20260925 00:57:53.251097 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 7438
I20260925 00:57:53.256186 17192 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPY1ZZ/build/debug/bin/kudu with pid 7513
2026-09-25T00:57:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3086 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6114 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282066 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST